System and method for assessing wound

ABSTRACT

The wound assessing method and system provide a convenient, quantitative mechanism for diabetic foot ulcer assessment.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 15/239,486, filed on Aug. 17, 2016, and entitled SYSTEM ANDMETHOD FOR ASSESSING WOUND, which claims priority to and benefit of U.S.Provisional Application No. 62/206,353, entitled AN AUTOMATIC ASSESSMENTSYSTEM OF DIABETIC FOOT ULCERS BASED ON WOUND AREA DETERMINATION, COLORSEGMENTATION AND HEALING SCORE EVALUATION, filed on Aug. 18, 2015, andof U.S. Provisional Application No. 62/375,225, entitled SYSTEM ANDMETHOD FOR ASSESSING WOUND, filed on Aug. 15, 2016, and is also acontinuation in part of U.S. application Ser. No. 14/528,397, entitledSYSTEM AND METHOD FOR ASSESSING WOUND, filed on Oct. 30, 2014, which inturn claims priority to and benefit of U.S. Provisional Application No.61/897,559, entitled SYSTEM AND METHOD FOR ASSESSING WOUND, filed onOct. 30, 2013, and U.S. Provisional Application No. 61/898,907, entitledSYSTEM AND METHOD FOR ASSESSING WOUND, filed on Nov. 1, 2013, all ofwhich are incorporated by reference herein in their entirety and for allpurposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under Grant No.IIS-1065298, awarded by the National Science Foundation (NSF). Thefederal government may have certain rights in the invention.

BACKGROUND

The present teachings relate to a system and a method for assessingchronic wound. More particularly, the present teachings relate to asystem and a method for assessing wound for patients with, for example,type 2 diabetes and diabetic foot ulcers. One way to assess wound is touse a specialized camera to capture the wound image, then calculates thewound area and organizes wound images from different patients and storesimages in a central location. Another way to assess wound is to use amobile wound analyzer (MOWA), which is an Android-based software,intended for smart phones and tablets, for analysis of wound images. Thewound boundary needs to be traced manually after which the softwarecalculates the wound area and performs color analysis within the woundboundaries.

The conventional art does not address the problem of capturing footimages when the patients with diabetes have limited mobility. Inaddition, the prior art device is very costly and not affordable forindividual patients to own, apart from MOWA, which, however, is designedfor clinicians. Further, the prior art is not designed for joint use byboth the patient and his/her doctor, through automatic upload of raw andanalyzed wound images to cloud storage for easy access by the physician.Accordingly, there is a need to develop new system and method forassessing wound that overcome the above drawbacks in the prior art.

There is a desire and need for systems designed to operate in anunconstrained setting where the caregiver (nurse, technician and woundspecialist) simply captures an image of the wound with a handhelddevice, without needing to record the distance to the wound or the anglebetween the optical path and the wound surface. In addition, thedistance, the angle as well as the lighting conditions can be expectedto vary from one patient visit to the next. However, in order to measurethe wound area in absolute terms (say, mm²) the distance and angle mustbe known, and color correction must also be introduced to correct forchanges to the color spectrum of the light. There is a need for systemsand methods for correcting the wound area if the image was acquired atan angle relative to normal incidence.

BRIEF SUMMARY

The present teachings provide patients with diabetes and chronic footulcers an easy-to-use and affordable tool to monitor the healing oftheir foot ulcers via a healing score; at the same time, the patient'sphysician can review the wound image data to determine whetherintervention is warranted. The system is also applicable for patientswith venous leg ulcers. The system and method also includes correctingthe wound area if the image was acquired at an angle relative to normalincidence.

In accordance with one aspect, the present teachings provide a methodfor assessing wound. In one or more embodiments, the method of theseteachings includes capturing an image of a body part including the woundarea, analyzing the image to extract a boundary of the wound area,performing color segmentation within the boundary, wherein the woundarea is divided into a plurality of segments, each segment beingassociated with a color indicating a healing condition of the segmentand evaluating the wound area.

In accordance with another aspect, the present teachings provide asystem for assessing wound. In one or more embodiments, the system ofthese teachings includes an image acquisition component configured tocapture an image of a body part including a wound area, an imageanalysis module configured to extract a boundary of the wound area; animage segmentation module configured to perform color segmentationwithin the boundary of the wound area, wherein the wound area is dividedinto a plurality of segments, each segment being associated with a colorindicating a healing condition of the segment and a wound evaluationmodule configured to evaluate the wound area.

In accordance with a further aspect, the present teachings provide amethod for assessing wounds including correcting the wound area if theimage was acquired at an angle relative to normal incidence, where themethod includes capturing an image of a body part including a wound areaand of a calibration patch, the calibration patch located proximate tothe wound area and substantially in a same plane as the wound area, thecalibration patch comprising a number of concentric, substantiallycircular areas, segmenting the image, determining a boundary of thewound area, determining a calibration patch area and the number ofconcentric, substantially circular areas, determining, from thecalibration patch area and the number of concentric, substantiallycircular areas, whether the image was acquired at an angle relative tonormal incidence, performing color segmentation within the boundary,wherein the wound area is divided into a plurality of segments, eachsegment being associated with a color indicating a healing condition ofthe segment, correcting, when the image was acquired at the anglerelative to normal incidence, the wound area and evaluating the woundarea.

In accordance with a yet another aspect, the present teachings provide asystem for assessing wounds including correcting the wound area if theimage was acquired at an angle relative to normal incidence, where thesystem includes an image acquisition device configured for capturing animage of a body part including a wound area and of a calibration patch,the calibration patch located proximate to the wound area andsubstantially in a same plane as the wound area, the calibration patchcomprising a number of concentric, substantially circular areas, and oneor more processors configured to: segment the image, determine aboundary of the wound area, determine a calibration patch area and thenumber of concentric, substantially circular areas, determine, from thecalibration patch area and the number of concentric, substantiallycircular areas, whether the image was acquired at an angle relative tonormal incidence, perform color segmentation within the boundary,wherein the wound area is divided into a plurality of segments, eachsegment being associated with a color indicating a healing condition ofthe segment, correct, when the image was acquired at the angle relativeto normal incidence, the wound area and evaluate the wound area.

A number of other embodiments of the method and system of theseteachings are presented herein below.

For a better understanding of the present teachings, together with otherand further needs thereof, reference is made to the accompanyingdrawings and detailed description and its scope will be pointed out inthe appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 represents a schematic block diagram representation of oneembodiment of the system of these teachings;

FIG. 1a represents a schematic block diagram representation of onecomponent of an embodiment of the system of these teachings;

FIGS. 1b-1i show exemplary embodiments of images acquired with anembodiment of the system of these teachings;

FIG. 2a is a flowchart representation of the mean shift basedsegmentation algorithm as used in these teachings;

FIG. 2b is a flowchart representation of the mean shift based method forwound boundary determination as used in these teachings;

FIGS. 3a-3d are results of different operations in the wound boundarydetermination of these teachings;

FIGS. 4a-4c are results in the calculation of wound location of theseteachings;

FIG. 5 is a flowchart representation of the machine learning basedmethod for wound boundary recognition of these teachings;

FIG. 5a is a flowchart representation of the TextonBoost process used inthe CRF based method for wound boundary recognition of these teachings;

FIG. 5b is a flowchart representation of the CRF based method for woundboundary recognition of these teachings;

FIG. 6 shows exemplary results of a wound image obtained by a majorityvote scheme as used in these teachings;

FIG. 7 shows a comparison of original images, images obtained by themajority vote scheme and wound recognition using the machine learningmethods of these teachings;

FIG. 8 shows the conventional confusion matrix that is used in theseteachings;

FIG. 9a is a flowchart representation of the K-means algorithm used inthese teachings;

FIG. 9b is a flowchart representation of Color segmentation method ofthese teachings;

FIGS. 10a-10i show images of wound areas and all results of colorsegmentation methods used in these teachings;

FIG. 10j shows software interface screenshot for presenting wound imagesto clinicians in one embodiment of these teachings; clinicians click the“Next Image” button to view the next image for current patient, clickthe “Next Patient” button to score the images for the next patient andclick “Next Phase” button to score all images again with different giveninformation;

FIG. 11 is a schematic representation of a component of one embodimentof the system of these teachings;

FIGS. 12a-12c are graphical representations of a component of oneembodiment of the system of these teachings;

FIG. 13 is a schematic representation of front surface mirrors as usedin one component of the system of these teachings;

FIG. 14 is a schematic block diagram representation of anotherembodiment of the system of these teachings;

FIG. 15 is a schematic block diagram presentation of another componentof a further embodiment of the system of these teachings; and

FIG. 16 is a flowchart representation of the use of a further embodimentof the system of these teachings.

DETAILED DESCRIPTION

The following detailed description is not to be taken in a limitingsense, but is made merely for the purpose of illustrating the generalprinciples of these teachings, since the scope of these teachings isbest defined by the appended claims. Although the teachings have beendescribed with respect to various embodiments, it should be realizedthese teachings are also capable of a wide variety of further and otherembodiments within the spirit and scope of the appended claims.

As used herein, the singular forms “a,” “an,” and “the” include theplural reference unless the context clearly dictates otherwise.

Except where otherwise indicated, all numbers expressing quantities ofingredients, reaction conditions, and so forth used in the specificationand claims are to be understood as being modified in all instances bythe term “about.”

In the following, the term “handheld mobile communication device,” asused herein, refers to a device capable of being handheld and ofexecuting applications, and which is portable. In one instance, themobile communication device has one or more processors and memorycapability. Examples of mobile communication devices, these teachingsnot being limited to only these examples, include smart mobile phones,digital personal assistants, etc.

The present teachings relate to a wound image analysis system, which maybe implemented as hardware and/or software. In one embodiment, the woundimage analysis system of the present teachings is designed to operate ona handheld mobile communication device, such as a smart phone. The woundimage analysis system may be used in private homes or elder carefacilities by the patient him/herself, or in collaboration with acaregiver, with the relevant image data automatically uploaded to securecloud storage, to be accessible for perusal by the patient's doctorand/or clinicians in the patient's wound clinic. An alert system cannotify the patient's doctor if wound data exceeds some preset bounds. Inanother embodiment, the wound image analysis system of the presentteachings may operate in a wound clinic and cover several patients. Inthis embodiment, a smart phone is use collaboratively with a laptop(i.e., a smart phone-laptop collaborative system).

In one embodiment, the wound image analysis method of the presentteachings includes the following main parts: (i) image preprocessing,(ii) method for determining the wound boundary, (iii) method for colorimage segmentation, (iv) method for computing the healing score. Inother embodiments, the system of these teachings component configured todetermine the wound boundary, configured to perform color imagesegmentation and component configured to assess the wound area. Otherembodiments of the system of these teachings also include an imagecapture box to aid the patient and/or his/her caregiver in capturingimages of the foot ulcer under controlled distance and light conditions,and cloud storage and clinical access solution. Each of these componentswill be described briefly below, with additional details given in theattached documents. While each system component is essential for thefunctionality of the system, not all components are necessary to operatethe wound image analysis system.

(i) Image pre-processing. A JPEG image captured by a smart phone isconverted into an RGB bitmap image. An image noise reduction filter isapplied to down-sample the image for faster processing.

(ii) Component configured to determine the wound boundary. The woundboundary detection method is based on the mean shift segmentation of thewound image. The method first detects the outline of the foot and thenwithin the boundary of the foot locates the outline of the wound. A moreaccurate method may be used for wound boundary detection based on skillsand insight by experienced wound clinicians. For this purpose, machinelearning methods, such as the Support Vector Machine, may be used totrain the wound analysis system to learn about the essential featuresabout the wound. However, there may be concerns about the robustness ofthe Support Vector Machine method. A Conditional Random Field basedmodel for wound boundary detection is provided herein below.

(iii) Component configured for color image segmentation. The colorsegmentation method is instrumental in determining the healing state ofthe wound where red indicates healing, yellow indicates inflamed, andblack indicates necrotic.

(iv) Component configured to compute a healing score. The Healing Scoreis an important element of communicating in a simple fashion the healingstatus of the patient's wound. The Healing Score is a weighted sum offactors, such as: wound area; weekly change in wound area; woundtexture; relative size and shapes of the healing, inflamed and necroticregions within the wound boundary, and possibly the skin color aroundthe wound. The weighing factors are determined from expert clinicalinput.

(v) Image capture box. The image capture box is a device that allows apatient, possibly with the aid of his/her caregiver, to both visuallyobserve the appearance of a wound on the sole of the foot as well ascapture an image of the wound for storage and analysis. It is a compactbox, where the patient's foot can rest comfortably on a 450 angledsurface next to the smart phone holder. The angled surface can readilybe flipped to accommodate right foot as well as left foot. The boxcontains two front surface mirrors and warm white LED lighting.

(vi) Cloud storage and clinical access solution. The cloud storage andclinical access solution automatically uploads relevant wound data tothe cloud (e.g., network accessible storage) from the smart phone,either utilizing Wi-Fi (802.11), 3G, or other wireless network. Relevantdata comprises wound image data, which is automatically uploaded inencrypted form to secure cloud storage, to be accessible for perusal bythe patient's doctor. An alert system can alert the doctor if wound dataexceeds some preset bounds.

In another embodiment, the wound image analysis system operates in awound clinic and covers several patients. In this embodiment, a handheldmobile communication device-Computing Component collaborative system isused, in which a captured image is automatically transferred to acomputing component. In one instance, the transfer occurs through apeer-to-peer based Wi-Fi system or Local Area Network, using a wired orwireless router.

Moreover, in a further embodiment, instead of using the smart phonecamera as the image acquisition device, the wound image analysis systemcan use a compact hyperspectral camera integrated into the image capturebox. In one instance, three types of LED illumination are integratedinto the image capture box: infrared (IR) LED illumination; visiblelight illumination, using the already built-in warm white LEDillumination; and ultraviolet (UV) LED illumination. This allows thewound to be imaged by three distinct wavelength bands, with theexpectation of revealing much better diagnostic information about thewound. In one instance, The hyperspectral camera includes directcommunication capability, such as but not limited to Wi-Fi, by which thecaptured images are transmitted to a device, such as a handheld mobilecommunication device or a computing device, for processing and cloudupload.

In accordance with one aspect, the present teachings provide a methodfor assessing wound. In one or more embodiments, the method of theseteachings includes capturing an image of a body part including a woundarea, analyzing the image to extract a boundary of the wound area,performing color segmentation within the boundary, wherein the woundarea is divided into a plurality of segments, each segment beingassociated with a color indicating a healing condition of the segmentand evaluating the wound area.

In one or more embodiments, the system of these teachings includes animage acquisition component configured to capture an image of a bodypart including a wound area, an image analysis module configured toextract a boundary of the wound area; an image segmentation moduleconfigured to perform color segmentation within the boundary of thewound area, wherein the wound area is divided into a plurality ofsegments, each segment being associated with a color indicating ahealing condition of the segment and a wound evaluation moduleconfigured to evaluate the wound area.

In accordance with another aspect, the present teachings provide amethod for assessing wounds including correcting the wound area if theimage was acquired at an angle relative to normal incidence, where themethod includes capturing an image of a body part including a wound areaand of a calibration patch, the calibration patch located proximate tothe wound area and substantially in a same plane as the wound area, thecalibration patch comprising a number of concentric, substantiallycircular areas, locating the calibration patch in the image, segmentingthe calibration patch in the image, the image, determining a boundary ofthe wound area, determining a calibration patch area and the number ofconcentric, substantially circular areas, determining, from thecalibration patch area and the number of concentric, substantiallycircular areas, whether the image was acquired at an angle relative tonormal incidence, performing color segmentation within the boundary,wherein the wound area is divided into a plurality of segments, eachsegment being associated with a color indicating a healing condition ofthe segment, correcting, when the image was acquired at the anglerelative to normal incidence, the wound area and evaluating the woundarea.

In one or more other embodiments, the present teachings provide a systemfor assessing wounds including correcting the wound area if the imagewas acquired at an angle relative to normal incidence, where the systemincludes an image acquisition device configured for capturing an imageof a body part including a wound area and of a calibration patch, thecalibration patch located proximate to the wound area and substantiallyin a same plane as the wound area, the calibration patch comprising anumber of concentric, substantially circular areas, and one or moreprocessors configured to: segment the image, determine a boundary of thewound area, determine a calibration patch area and the number ofconcentric, substantially circular areas, determine, from thecalibration patch area and the number of concentric, substantiallycircular areas, whether the image was acquired at an angle relative tonormal incidence, perform color segmentation within the boundary,wherein the wound area is divided into a plurality of segments, eachsegment being associated with a color indicating a healing condition ofthe segment, correct, when the image was acquired at the angle relativeto normal incidence, the wound area and evaluate the wound area.

One embodiment of the system of these teachings is shown in FIG. 1.Referring to FIG. 1, in the embodiment shown there in, an image capturecomponent 15 captures the image, where the image can include the imageof a body part including the one area and a calibration patch, the imageis preprocessed and provided to an image segmentation component 27 and awound boundary determination component 29 that are configured to extracta boundary of the area of the wound. A color image segmentationcomponent 35 is configured to perform color segmentation on the imagewithin the wound boundary. In the color segmentation the area of theimage within the one boundary is divided into a number of segments, eachsegment being associated with the color that indicates a healingcondition of the segment. A wound evaluation component 45 receives theinformation from the image segmentation component 35 and provides ananalysis of the wound healing trend.

In one instance, after the wound image is captured, the JPEG file pathof this image is added into a wound image database. This compressedimage file, which cannot be processed directly with the main imageprocessing algorithms, therefore needs to be decompressed into a 24-bitbitmap file based on the standard RGB color model. In one instance, thebuilt-in APIs of the smartphone platform to accomplish the JPEGcompression and decompression task. The “image quality” parameter isused to control the JPEG compression rate. In one embodiment, setting“image quality” to 80 was shown empirically to provide the desirablebalance between quality and storage space. In that embodiment, for anefficient implementation on the smartphone alone, no method was used tofurther remove the artifacts introduced by JPEG lossy compression.

In one instance, in the Image preprocessing step, the high resolutionbitmap image is first down-sampled to speed up the subsequent imageanalysis and to eliminate excessive details that may complicate thewound image segmentation. In one instance, the original image (pixeldimensions 3264×2448) is down-sampled by a factor 4 in both thehorizontal and vertical directions to pixel dimensions of 816×612, whichhas proven to provide a good balance between the wound resolution andthe processing efficiency. Afterwards, the images is smoothed to removenoise (assumed mainly to be Gaussian noise produced by image acquisitionprocess) by using the Gaussian blur method whose standard deviationσ=0.5 was empirically judged to be substantially optimal based onmultiple experiments.

In one or more instances, in the method of these teachings, analyzingthe image includes performing mean shift segmentation and objectrecognition and, in the system of these teachings, the image analysiscomponent is configured to perform mean shift segmentation and objectrecognition. In the mean shift based image segmentation and region mergeoperations, the wound boundary determination task doesn't rely on anyclinical inputs. The Foot outline detection is accomplished by findingthe largest connected component in the segmented image. Afterwards, awound boundary determination was carried out based on the smart analysisof previous foot outline detection result. This solution is veryefficient and easy to be implemented on the handheld mobilecommunication device platform. However, the toe-amputation status has tobe recorded as part of patients' data.

Many non-parametric clustering methods can be separated into two parts:hierarchical clustering and density estimation. Hierarchical clusteringis a method of cluster analysis, which seeks to build a hierarchy ofclusters. Strategies for hierarchical clustering generally fall into twotypes including 1) agglomerative: this a “bottom up” approach in whicheach observation starts in its own cluster and pairs of clusters aremerged as one moves up the hierarchy; 2) divisive: this is a “top down”approach in which all observations start in one cluster and splits areperformed recursively as one moves down the hierarchy. On the otherhand, the concept of the density estimation-based non-parametricclustering method is that the feature space can be considered as theexperiential probability density function of the represented parameter.The mean shift algorithm can be classified as density estimation. Itadequately analyzes feature space to cluster them and can providereliable solutions for many vision tasks.

In general, the mean shift algorithm models the feature vectorsassociated with each pixel (e.g., color and position in the image grid)as samples from an unknown probability density function ƒ(x) and thentry to find clusters (dense areas) in this distribution. The key to meanshift is a technique for efficiently finding peaks in thishigh-dimensional data distribution (In these teachings, there will be 5dimension including 3 color range dimension and 2 spatial dimension)without ever computing the complete function explicitly. The problem issimplified to how to find the local maxima (peaks or modes) in anunknown density distribution.

Let us take a look at the kernel density estimation definition at first.Given n data points x_(i), =1, . . . , n in the d-dimensional spaceR^(d), the multivariate kernel density estimator with kernel K(x) isshown as below (see D. Comaniciu, P. Meer, Mean Shift: A Robust ApproachToward Feature Space Analysis, IEEE Tran. on Pattern Analysis andMachine Intelligence, vol 24 (5), May 2002, pp. 603-619, which isincorporated by reference herein is entirety and for all purposes).

$\begin{matrix}{{f(x)} = {\frac{1}{{nh}^{d}}{\sum\limits_{i = 1}^{n}{K\left( \frac{x - x_{i}}{h} \right)}}}} & {{Eq}.\mspace{14mu} 3.1}\end{matrix}$where h is one bandwidth parameter satisfying h>0 and K is the radiallysymmetric kernels satisfyingK(x)=c _(k,d) k(∥x∥ ²)  Eq. 3.2where c_(k,d) is a normalization constant which makes K(x) integrate toone. The function k(x) is the profile of the kernel, defined only forx≥0. After applying the profile notation, the density estimator in Eq.3.1 can be written as below.

$\begin{matrix}{{f_{h,K}(x)} = {\frac{c_{k,d}}{{nh}^{d}}{\sum\limits_{i = 1}^{n}{k\left( {\frac{x - x_{i}}{h}}^{2} \right)}}}} & {{Eq}.\mspace{14mu} 3.3}\end{matrix}$

In mean shift algorithm, a variant of what is known in the optimizationliterature is used as multiple restart gradient descent. Starting atsome guess for a local maxima y_(k), which can be a random input datapoint x_(i), mean shift computes the density estimate ƒ(x) at y_(k) andtake a uphill step in the gradient descent direction. The gradient ofƒ(x) is given by

$\begin{matrix}{{\nabla{f(x)}} = {\frac{2c_{k,d}}{{nh}^{d + 2}}{\sum\limits_{i = 1}^{n}{\left( {x_{i} - x} \right){g\left( {\frac{x - x_{i}}{h^{2}}}^{2} \right)}}}}} & {{Eq}.\mspace{14mu} 3.4}\end{matrix}$where g(r)=−k′(r) and n is the number of neighbors taken into account inthe 5 dimension sample domain. In one instance, the Epanechinikov kernelshown as Eq. 3.2 is used, which makes the derivative of this kernel is aunit sphere.If the Eq. 3.4 is rewritten as the following

$\begin{matrix}{{\nabla{f(x)}} = {{\frac{2c_{k,d}}{{nh}^{d + 2}}\left\lbrack {\sum\limits_{i = 1}^{n}{g\left( {\frac{x - x_{i}}{h}}^{2} \right)}} \right\rbrack} \times {m(x)}}} & {{Eq}.\mspace{14mu} 3.5}\end{matrix}$

$\begin{matrix}{{m(x)} = {\frac{\sum\limits_{i = 1}^{n}{x_{i}{g\left( {\frac{x - x_{i}}{h}}^{2} \right)}}}{\sum\limits_{i = 1}^{n}{g\left( {\frac{x - x_{i}}{h}}^{2} \right)}} - x}} & {{Eq}.\mspace{14mu} 3.6}\end{matrix}$

The vector in Eq. 3.6 is called the mean shift vector, since it is thedifference between the weighted mean of the neighbors x_(i) around x andthe current value x. In the mean-shift procedure, the current estimateof the mode y_(k) at iteration k is replaced by its locally weightedmean as shown below:y _(k+1) =y _(k) +m(y _(k))  Eq. 3.7

This iterative update of the local maxima estimation will be continueduntil the convergence condition is met. In one instance, the convergencecondition is set as the Euclidean length of the mean shift vector issmaller than a preset threshold.

Actually, in the mean shift based algorithm as used in these teachings,the mean shift update thread is performed multiple times by taking eachpixel in the image plane as the starting point and replace the currentpixel with the converged local maxima point. All the pixels leading tothe same local maxima will be set as the same label in the label image.After this, the very first mean shift segmentation (strictly speaking,it is the mean shift smooth filtering) result is obtained while it isalmost definitely over-segmented. Therefore, the over-segmented imagehas to be merged based on some rules. In the fusion step, extensive usewas made of region adjacency graphs (RAG).

The method flowchart is shown as in FIG. 2a . The reason for using theLUV color space is because that perceived color differences shouldcorrespond to Euclidean distances in the color space chosen to representthe features (pixels). The LUV and LAB color space were especiallydesigned to best approximate uniformly color space. To detect all thesignificant modes, the following basic mean shift filtering processshould be run for multiple times (evolving in principle in parallel)with different starting points that cover the entire feature space. Inthese teachings, all the pixels in the image domain are used as thestarting points.

The first step in the mean shift based feature space with the underlyingdensity ƒ(x) is to find the modes of this density. The modes are locatedamong the zeros of the gradient, ∇ƒ(x)=0, and the mean shift procedureis an elegant way to locate these zeros without estimating the densitycompletely. The mean shift vector m(x) computed in Eq. 3.6 with kernel gis proportional to the normalized density gradient estimate obtainedwith kernel k. In Eq. 3.6, “n” represents the number of neighbor pixelsx_(i) involved in the kernel density estimation (see, for example, C. M.Christoudias, B. Georgescu, P. Meer, Synergism in Low Level Vision, IEEEProc. of 16^(th) Inter. Conf. on Pattern Recognition, 2002. Vol. 4: pp.150-155, which is incorporated by reference herein is entirety and forall purposes), All the neighbor pixels located within the Euclideandistance h from the current pixel will be chosen. The mean shift vectorthus always points toward the direction of the maximum increase in thedensity. In this case, the y_(k) is iteratively updated according to Eq.3.7 until the convergence will lead to the local maxima for the currentpoint in the probability density function (PDF). The convergence isdefined as when the difference between y_(k) and y_(k+1). is smallerthan a specified threshold value.

In Eq. 3.6, “i” represents the i^(th) gradient descent path. After allthe local maxima have been detected from different starting points, allthe points on the path leading to each maxima will be claimed to belongto the basin marked by the current maxima. Then the basins with the sizesmaller than the pre-stetting threshold value will be merged to thenearest basin whose size is bigger than a preset threshold. In bothequations, the pixel is described by a 5 dimension vector concatenatedin the joint spatial-range domain including 3 elements for the LUV colordomain and 2 elements for the spatial domain. As stated hereinabove, thekernel function k is chosen as the Epanechinikov kernel. In theseteachings, the combined kernel function shown in Eq. 3.8 is used.

$\begin{matrix}{{K_{{hs},{hr}}(x)} = {\frac{C}{{hs}^{2}{hr}^{3}}{k\left( {\frac{x^{s}}{h^{s}}}^{2} \right)}{k\left( {\frac{x^{r}}{h^{r}}}^{2} \right)}}} & {{Eq}.\mspace{14mu} 3.8}\end{matrix}$where hs and hr are different bandwidth values for spatial domain andrange domain, respectively.

After the initial mean shift filtering procedure, the over-segmentedimage are merged based on some rules. In the fusion step, extensive usewas made of region adjacency graphs (RAG) (see, for example, C. M.Christoudias, B. Georgescu, and P. Meer, “Synergism in low levelvision,” Object Recognit. Support. by user Interact. Serv. Robot., vol.4, no. 2, pp. 150-155, 2002, or A. Duarte, Á. Sánchez, F. Fernández, andA. S. Montemayor, “Improving image segmentation quality througheffective region merging using a hierarchical social metaheuristic,”Pattern Recognit. Lett., vol. 27, no. 11, pp. 1239-1251, 2006, both ofwhich are incorporated by reference herein in their entirety and for allpurposes). The initial RAG was built from the initial over segmentedimage, the modes being the vertices of the graph and the edges weredefined based on 4-connectivity on the lattice. The fusion was performedas a transitive closure operation on the graph, under the condition thatthe color difference between two adjacent nodes should not exceed hr/2.

In other embodiments, the image of a body part includes a wound area andof a calibration patch, the calibration patch located proximate to thewound area and substantially in a same plane as the wound area, thecalibration patch comprising a number of concentric, substantiallycircular areas. “Substantially” is used in the description of thelocation of the calibration patch since body parts are not absolutelyplanar and the calibration patch is located approximately enough to thewound area so that to an acquisition device the planar difference iswithin the uncertainty of planarity measurement of the image acquisitiondevice.

In accordance with a further aspect, the present teachings provide amethod for assessing wounds including correcting the wound area if theimage was acquired at an angle relative to normal incidence, where themethod includes capturing an image of a body part including a wound areaand of a calibration patch, the calibration patch located proximate tothe wound area and substantially in a same plane as the wound area, thecalibration patch comprising a number of concentric, substantiallycircular areas, segmenting the image, determining a boundary of thewound area, determining a calibration patch area and the number ofconcentric, substantially circular areas, determining, from thecalibration patch area and the number of concentric, substantiallycircular areas, whether the image was acquired at an angle relative tonormal incidence, performing color segmentation within the boundary,wherein the wound area is divided into a plurality of segments, eachsegment being associated with a color indicating a healing condition ofthe segment, correcting, when the image was acquired at the anglerelative to normal incidence, the wound area and evaluating the woundarea.

In accordance with a yet another aspect, the present teachings provide asystem for assessing wounds including correcting the wound area if theimage was acquired at an angle relative to normal incidence, where thesystem includes an image acquisition device configured for capturing animage of a body part including a wound area and of a calibration patch,the calibration patch located proximate to the wound area andsubstantially in a same plane as the wound area, the calibration patchcomprising a number of concentric, substantially circular areas, and oneor more processors configured to: segment the image, determine aboundary of the wound area, determine a calibration patch area and thenumber of concentric, substantially circular areas, determine, from thecalibration patch area and the number of concentric, substantiallycircular areas, whether the image was acquired at an angle relative tonormal incidence, perform color segmentation within the boundary,wherein the wound area is divided into a plurality of segments, eachsegment being associated with a color indicating a healing condition ofthe segment, correct, when the image was acquired at the angle relativeto normal incidence, the wound area and evaluate the wound area.

In one exemplary embodiment, the circular calibration patch has of anouter black ring, a middle yellow ring and a red circle in the center ofthe patch. The areas of the black ring, the yellow ring and the redcircle are identical. The diameter of the patch can range from 10 mm to20 mm, but is typically around 15 mm. The calibration patch must beplaced near the wound and in the same plane as the wound. To locate thepatch and to find the patch boundaries, a color patch detection methodis applied.

A flow diagram for the color patch detection method is shown in the FIG.1a . Since some of the steps in the flow diagram of FIG. 1a are the sameas in FIG. 1, the same identify numbers are used. Referring to FIG. 1a ,an image of a body part including a wound area and of a calibrationpatch is captured (step 15), the calibration patch located proximate tothe wound area and substantially in a same plane as the wound area, thecalibration patch comprising a number of concentric, substantiallycircular areas. The image is segmented (step 27) in the mean shift basedalgorithm as used in these teachings, the mean shift update thread isperformed multiple times by taking each pixel in the image plane as thestarting point and replace the current pixel with the converged localmaxima point. All the pixels leading to the same local maxima will beset as the same label in the label image. After this, the very firstmean shift segmentation (strictly speaking, it is the mean shift smoothfiltering) result is obtained while it is almost definitelyover-segmented. Therefore, the over-segmented image has to be mergedbased on some rules (RAG). The image is analyzed using the followingcriteria (step 55):

1) The mean color value is “similar enough” to the red value.norm(Vc−Vr)<threshold

-   -   where Vc represents the mean color value for a certain region,        Vr represents the standard red color value which is determined        empirically. The threshold is also determined empirically.

2) The uniformity in the region is “small enough”Uc<threshold

-   -   where the uniformity measure is given in the equation below

$U_{\alpha} = {1 - \frac{2{\sum\limits_{R_{j} \in \alpha}{\sum\limits_{i \in R_{i}}{{f_{i} - \overset{\_}{f_{j}}}}}}}{\sum\limits_{R_{j} \in \alpha}{A_{j}{{f_{\max} - f_{\min}}}}}}$

-   -   in which    -   f_(i) is the color vector in CIE lab space for pixel i in region        R_(j), f_(j) is the average value of all f_(i) in R_(j). A_(j).        is the area of the region R_(j), f_(min), and f_(max) are the        maximum and minimum values in this region. (See F. Veredas, H.        Mesa, and L. Morente, “Binary tissue classification on wound        images with neural networks and bayesian classifiers,” IEEE        Trans. Med. Imaging, vol. 29, no. 2, pp. 410-427, 2010,        and M. D. Levine and A M. Nazif, “Dynamic measurement of        computer generated image segmentations,” IEEE Trans. Pattern        Anal. Mach. Intell., vol. 7, no. 2, pp. 155-164, 1985, both of        which are incorporated by reference herein in in their entirety        and for all purposes.)

The threshold is also determined empirically. The region satisfying thetwo criteria above will be claimed as the patch region (step 60). Ifnone of the regions satisfy both criteria, then we will choose theregion whose mean color is closest to the standard red color. In thiscase, we will relax the requirement for uniformity since we find outthat the uneven illumination may cause the patch color to be non-uniformin the image.

FIG. 1(b)-1(i) show examples of images of the wound area and acalibration patch.

After the patch has been located, an outer boundary of the patch isfitted to an elliptical shape (using, for example, but not limited to,the curve fitting toolbox in Matlab) and the minor and major axisdetermined. In one instance, if the difference between major and minoraxis is less than a predetermined threshold, the outer boundary of thepatch is fitted to a circular shape.

Under normal incidence, i.e., the optical path forms a right angle withthe wound surface, the patch will appear as a circle in the image, andin this case the wound dimensions scale linearly with the patchdiameter, allowing a simple calibration. When the image containing thewound and the calibration patch is acquired at an angle relative tonormal incidence, the wound will appear compressed in one dimension, andthe calibration patch will appear as an ellipse. This angle can becalculated from an inverse cosine operation on the ratio of the shortaxis to the long axis of the patch, and the wound area can now becorrected for the incident angle as well. Finally, the actual colors ofthe black ring, the yellow ring and the red circle, as they appear inthe image, become the calibration color for the color separation.

The incident angle Let ø be the incident angle (where under normalincidence, we have ø=0), determined as follows

$\phi = {\cos^{- 1}\left( \frac{{observed}\mspace{14mu}{minor}\mspace{14mu}{axis}\mspace{14mu}{of}\mspace{14mu}{patch}}{{observed}\mspace{20mu}{major}\mspace{14mu}{axis}\mspace{14mu}{of}\mspace{20mu}{patch}} \right)}$given that the patch will appear elliptical when observed at non-normalincidence.

If the image was acquired at the angle relative to normal incidence, thewound area is corrected. Let R be the reference range, which isestablished a priori as the distance where—in a wound image—the area ofa representative wound can be determined with the highest accuracy. Forclarification, in this document a pixel represents a linear dimensionwhile a sq. pixel represents an area. After R has been determined, theconversion factor, β, between the diameter of the calibration patch, inpixels, and the actual physical diameter, in mm, is established andstored. Further note that all dimensions will be given in pixels and allareas will be given in sq. pixels until a dimension or an area has beennormalized to the reference range, R.

The factor β will primarily be used for the determination of wounddimensions. The conversion of a given wound area in sq. pixels,A_(sqpx), measured at the reference range, R, to the wound area in mm²,A_(wound), is given as follows:A _(wound) =A _(sqpx)β².Under normal incidence of the camera lens to the wound line, the patchis a circle and we have ø=0. To determine the actual range, r, at whichthe image of the wound and the patch was acquired, we extract d_(max)for the patch (=diameter, for the case of ø=0), along with the a prioriknown d_(ref), defined as the diameter of the patch at the referencerange, R, in pixels. Given that

${\frac{r}{R} = \frac{d_{\max}}{d_{ref}}},$we find the range r as

$r = {R{\frac{d_{\max}}{d_{ref}}.}}$Define the area of a given wound, expressed in sq. pixels and observedat the reference range, R, as A_(ref). If A_(obs) is the observed woundarea, also expressed in sq. pixels, but observed at range r and incidentangle ø, then

$A_{obs} = {{A_{ref}\left( \frac{r}{R} \right)}^{2}\cos\;\phi}$

Note that this expression does not consider a curvature of the wound. Inan actual measurement, we solve for A_(ref) in the above equation. Butsince some algorithms may have angle and range dependent errors, theresult is referred to as the corrected wound area, or A_(corr), insteadof A_(ref).

$A_{corr} = {\frac{A_{obs}}{\cos\;\phi}\left( \frac{R}{r} \right)^{2}}$

The wound boundary determination approach using mean shift basedsegmentation is theoretically full-automatic and does not depend on anya priori manual input, which makes it computationally very economic andflexible for implementation in any hardware platforms. In FIG. 2b , theworkflow of this approach is provided.

The mean shift based algorithm is first applied to segment the originalwound image into a number of homogeneous regions. The mean shiftalgorithm is chosen over other segmentation methods, such as level setand graph cut based algorithms for several reasons. First, the meanshift algorithm takes into consideration the spatial continuity insidethe image by expanding the original 3D color range space to 5D space,including two spatial components, since direct classification on thepixels proved to be inefficient. Second, a number of accelerationalgorithms are available. Third, for both mean shift filtering andregion merge methods, the quality of the segmentation is easilycontrolled by the spatial and color range resolution parameters. Hence,the segmentation algorithm can be adjustable to different degrees ofskin color smoothness by changing the resolution parameters. Finally,the mean shift filtering algorithm is suitable for parallelimplementation since the basic processing unit is the pixel. In thiscase, the high computational efficiency of GPUs can be exploited, whichcan further improve the efficiency and achieve the real time woundassessment even on the smartphone-alone system.

After applying the mean shift algorithm, the image is usuallyover-segmented, which means that there are more regions in thesegmentation result than necessary for wound boundary determination. Tosolve this problem, the over-segmented image is merged into a smallernumber of regions which are more object-representative based on somerules. In the fusion step, extensive use was made of region adjacencygraphs (RAG). The initial RAG was built from the initial over-segmentedimage, the modes being the vertices of the graph and the edges definedbased on 4-connectivity on the lattice. The fusion was performed as atransitive closure operation on the graph, under the condition that thecolor difference between two adjacent nodes should not exceed h_(ƒ),which is regarded as the region fusion resolution. Based on experimentalresults, the over-segmentation problem is found to be effectively solvedby region fusion procedure.

The wound boundary determination method is based on two assumptions.First, the foot image contains little information not related to thechronic wound. In reality, it is not a critical problem as it is assumedthat the patients and/or caregivers will observe the foot image with thewound on the smartphone screen before the image is captured to ensurethat the wound is clearly visible. Second, it is assumed that thehealthy skin on the sole of the foot is a nearly uniform color feature.

The largest connected component detection is first performed on thesegmented image, using the fast largest connected component detectionmethod including two passes. In the processing step Foot ColorThresholding, the color feature, extracted in the mean shiftsegmentation algorithm of this component, is compared with an empiricalskin color feature by calculating the Euclidean distance between thecolor vector for the current component and the standard skin colorvector from the Macbeth color checker. If the distance is smaller than apre-specified and empirically determined threshold value, the foot areais considered as having been located. Otherwise, the largest componentdetection algorithm is iteratively repeated on the remaining part of theimage while excluding the previously detected components until the colorthreshold condition is satisfied. After the foot area is located, abinary image is generated with pixels that are part of the foot labeled“1” (white) and the rest part of the image labeled “0” (black).

Then, the wound boundary determination tasks have to be classified intotwo categories: 1) the wound is fully enclosed within the foot outline;2) the wound is located at (or very near to) the boundary of the footoutline. The initial idea was to use the foot boundary smoothness todistinguish between these two situations. However, the problem is that agold standard for the ordinary smooth foot curve may be needed, i.e. theboundary of the healthy foot, and quantitatively compare the actuallydetected foot outline to it in some way. The search for such a groundtruth healthy foot curve is never an easy task. Moreover, it has to beensured that the patient's entire foot is imaged completely, which is adifficult-to-meet expectation for a self-management wound analysissystem considering the possible low mobility and the lack of experienceof handheld communication device use for most type 2 diabetic patients.Therefore, the following method is used to realize the taskclassification.

At first, one of the image morphology operations called a closingoperation (with a 9×9 circle structure element) is applied to remove allthe holes in the foot region (white part in the binary image) and smooththe external foot boundary, which will help us to eliminate the possibleinterference for accurate wound boundary determination. Secondly, thecombined region and boundary algorithm is applied to trace the externalfoot boundary along the edge of the white part in the foot binary image,as well as all the internal boundaries if there are any. For all theinternal boundaries in a foot region, only the ones with the perimeterlarger than a preset threshold (in one embodiment, it is set as 50 pixellengths) are kept. This simple threshold method may not be a perfectalgorithm but it works for most of the wound images in many instances.In other words, if there is at least one internal boundary exceeding thepreset threshold within the foot region, it is regarded as the woundboundary and returns it as the final boundary determination result. Onthe other hand, if there are not any internal boundaries whose lengthare beyond the threshold, other boundary determination methods may beneeded. Note that here it is assumed there is at least one wound area onthe photographed foot.

After a careful study and observation, a wound boundary determinationmethod, as shown in the right column in FIG. 2b , is used that isapplicable for a wound located at or near to the foot outline. As statedherein above, the external boundary of the non-enclosed foot outline isalready available.

As illustrated in the block diagram in FIG. 2b , the input to thismethod is the external boundary. Instead of keeping all the points onthe foot boundary, the edge points (all the points on the externalboundary of the foot region) are down-sampled by applying the HarrisCorner Detection method to a number of corner points (as shown in part(a) of FIG. 3). The corner points, also called junctions of edges, areprominent structural elements in an image and are therefore useful in awide variety of computer vision application. It is claimed that cornerpoints are more robust features for geometric shape detection than theregular edge points. In one instance, the perimeter of the foot outlineusually is made up of over 2000 pixels. After down-sampling by cornerdetection, the number of corner points is approximately around 60-80(also in terms of pixels). This will greatly improve the time efficiencyof the algorithm. Besides, this down-sampling procedure will alsobenefit us when detecting the turning points (this will be discussed indetail in herein below).

The third to the eighth blocks in the right column in FIG. 2b shows themain idea, which is to detect the three turning points on the footboundary. These turning points will be used to determine the woundsection on the foot outline (shown in the part (c) of FIG. 3 by markingthe three turning points with small black crosses). The turning pointscan be defined as the most abruptly changing points along the footboundary. After the three points are determined, one can move along fromthe global maximum point (which is the most concave point in the middle)to the two local minimum points along the foot boundary in two oppositedirections. Then the two local minimum points are connected by an arcwhich is the substantially optimal approximation to the non-closed partof the wound boundary. In one instance, an are is drawn from eitherlocal minimum to the other one with a radius equal to half of thediagonal length for the wound image. The trace generated by the abovetwo steps is the target wound boundary (as shown by a red curve in FIG.3d ).

For detecting the turning points, a maximum-minimum searching approachis used to detect the turning points. Herein below, a detaileddescription of this approach is provided.

First, all the corner points are sorted into a list based on theirposition on the foot boundary (from the top-right to top-left, in aclock-wise direction), then locate the two special extreme corner pointson the foot boundary: the leftmost and the rightmost (as indicated inpart (a) of FIG. 3). Then, the corner points are divided into twogroups: the corners points which located between the two extreme pointsand the corner points located outside this range. Note that thiscategorization is based on the clock-wise sorted corner points list. InFIG. 4a , the first group of corner points is marked by U and the secondgroup is marked by L. For the first group the vertical distance of eachcorner point to the top side of the Smallest Fitting Rectangle (SFR) ofthe foot region is calculated. The smallest fitting rectangle issupposed to be tangent to the foot area at four boundary points: thetop-most, bottom-most, left-most and right most point of the foot area,as shown by the frame in FIG. 4 b.

Similarly, the vertical distance of each corner point in the secondgroup to the bottom side of the SFR (as shown in FIG. 4b ) iscalculated. Afterwards, the turning points are located by seeking forthe corner point with global maximum vertical distance to thecorresponding rectangle side (top or bottom based on its group number:first or second) and also two corner points on each side of the maximumpoint with the smallest local minimum vertical distance (as shown FIG.3c ). The only concern is the search for target turning points may beaccidentally stopped by interfering local extrema, which is a commonproblem of most local search algorithms. As stated above, a certainnumber of corner points are kept on the foot outline. Based onexperimental results, it is found that this down-sampling procedure caneliminate most of the interfering local extrema which may impede thesearch for the optimal turning points.

The above disclosed method mainly classifies the wound locations intothree categories: 1) wound in the middle of the foot, 2) wound at theedge of the foot without toe-amputation and 3) wound at the edge of thefoot with toe-amputation. For the first category, the wound is supposedto be surrounded by healthy skin and can be easily detected by tracingthe internal boundary within the foot outline. For the second and thirdcategories, the three turning points detection method is applied tolocate the wound boundary which is assumed to be the most irregularlychanged section on the foot outline. In practice, the method dealingwith these two situations (with or without toe-amputation) is slightlydifferent. Hence, the toe-amputation information may need to be given asan input to the method and obtained as part of the patient's medicalinformation.

In another instance, in the method of these teachings, analyzing theimage includes using a trained classifier and, in the system of theseteachings, the image analysis component is configured to use a trainedclassifier.

A machine learning based solutions has been developed in which the woundboundary determination is an object recognition task since it is claimedthat the machine learning (ML) is currently the only known way todevelop computer vision systems that are robust and easily reusable indifferent environments. Herein below, the term “wound recognition” isused as the equivalent expression of “wound boundary determination”,since both have the same goal.

In object recognition field, three major tasks needed to be solved toachieve the best recognition performance: 1) find the bestrepresentation to distinguish the object and background, 2) find themost efficient object search method and 3) design the most effectivemachine learning based classifier to determine whether a representationbelongs to the object category or not.

For the chronic wound recognition method and components of theseteachings, a hybrid of the global window and local patch basedrepresentation which modifies the general form of the global texturedescriptors to be extracted within only local sub-windows or patches isused. A popular approach of this hybrid type is called Bags of VisualWords (BoW) which uses a visual vocabulary to compactly summarize thelocal patch descriptors within a region using a simple 1D histogram(see, for example: (i) Fei-Fei Li; Perona, P., “A Bayesian HierarchicalModel for Learning Natural Scene Categories”. 2005 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (CVPR'05). p. 524and (ii) Rob Fergus, Classical Methods for Object Recognition, slidespresented at ICCV 2009 course, both of which are Incorporated byreference herein in their entirety and for all purposes).

This representation is completely orderless, which means that greaterflexibility is allowed (for better or worse) with respect to viewpointand pose changes. At the same time, the invariance properties of theindividual local descriptors make them a powerful tool to tolerate thevariation of the viewpoint or pose while giving informative localappearance cues. The regularity or rigidity of an object category'sappearance pattern in 2D determines which style is better suited. Forexample, the class of frontal face is quite regular and similaritystructured across instances, and thus is more suitable for the 2Dlayout-preserving descriptors; in contrast, wounds represent a varietyof shapes of which most are irregular. This property makes it suited toa more flexible summary of the texture and key features. What isparticularly convenient about the bag-of-words (BOW) representation isthat it translates a (usually very large) set of high-dimensional localdescriptors into a single sparse vector of fixed dimensionality acrossall images. This in turn allows one to use many machine learningalgorithms that by default assume that the input space isvectorial—whether for supervised classification, feature selection, orunsupervised image clustering.

The object localization techniques can mainly fall into one of twocategories: 1) the “top-down” technique, which tries to fit a coarseglobal object model to each possible location on the image grid or 2)the “bottom-up” technique, which tries to produce a pixel levelsegmentation of the input image and are built from the bottom up onlearned local representation and can be seen as an evolution of texturedetectors. The sliding window technique is a typical example for thefirst category. Due to its algorithmic nature, the sliding window searchapproach suffers from several limitations including the highcomputational cost, little room for error and inflexible for accuratewound boundary determination. Hence, in some related references, it isclaimed the bottom-up technique is more suitable for object classsegmentation task (similar as the wound boundary determination task).

The supervised learning methods have been most widely used. Thisapproach will try to inferring a model from labeled training data. Inrelated references, the comparison of several most popular supervisedlearning methods is provided. Generally speaking, support vector machine(SVMs) tends to perform much better when dealing with multi-dimensionsand continuous features. (See, for example, Using Support Vector Machineas a Binary Classifier, International Conference on Computer Systems andTechnologies—CompSysTech' 2005 which is incorporated by reference hereinin its entirety and for all purposes). For SVMs, given a set of trainingexamples, each marked as belonging to one of two categories, an SVMtraining algorithm builds a model that assigns new examples into onecategory or the other. An SVM model is a representation of the examplesas points in space, mapped so that the examples of the separatecategories are divided by a clear gap that is as wide as possible. Newexamples are then mapped into that same space and predicted to belong toa category based on which side of the gap they fall on. In addition toperforming linear classification, SVMs can efficiently perform anon-linear classification using what is called the kernel trick, whichimplicitly mapping their linear inputs into high dimensional featurespaces.

In one embodiment of these teachings, a two-stage recognition schemebased on some object recognition approach already being successfullyapplied in the pedestrian recognition task is used. The workflow of thiswound recognition system is shown in FIG. 5.

In the training process, there are two stages. For both stages, the SVMbased binary classifier training method is used. In the first stage, thesuper-pixel segmentation is performed by either the mean shift or SLIC(simply linear iterative clustering) algorithm to group pixels intoperceptually meaningful atomic regions which can be used to replace therigid structure of the pixel grid. Then, the vector representation isbuilt up for each super-pixel by using the bag of words (BOW) histogrambased on local DSIFT (dense SIFT) or SURF feature descriptor within thecurrent super-pixel.

To generate this representation, the extracted descriptors are thenquantized using a K-means dictionary and aggregated into one normalizedhistogram h_(i)∈R₊ ^(K) for each super-pixel s_(i) in the image, where Kis the number of words predefined in the codebook (the set of clustersresulted from the K-means algorithm). In order to train a classifier,each super-pixel s_(i) is assigned the most frequent class label itcontains (in this case, some manually labeled ground truth images whichhave pixel-level granularity are needed). Then a SVM with an RBF kernelis trained on the labeled histograms for either category: wound andnon-wound. This yields discriminant functions is proposed in relativereferences and shown as below.

$\begin{matrix}{{C(h)} = {\sum\limits_{j = 1}^{L}\;{c_{i}{\exp\left( {{- \gamma}\;{d^{2}\left( {h,h_{i}} \right)}} \right)}}}} & (1)\end{matrix}$where c_(i)∈R are coefficients and h_(i) representative histograms(support vectors) selected by SVM training, γ∈R⁺ is a parameter selectedby cross-validation, and d²(h, h_(i)) is the vector distance between thecurrent histogram h and each support vector.

This classifier which results from this is very specific. It findssuper-pixels which resemble super-pixels that were seen in the trainingdata without considering the surrounding region. However, a drawback oftraining a classifier for each super-pixel is that the histogramsassociated with each super-pixel are very sparse, often containing onlya handful of nonzero-elements. This is due to the nature of thesuper-pixels: by definition they cover areas that are roughly similar incolor and texture. Since the features are fixed-scale and extracteddensely, the super-pixels sometimes contain tens or even hundreds ofdescriptors that quantize to the same visual word.

To overcome the problems caused by the lack of consideration of thesurrounding region of each super-pixel and sparse histogramrepresentation, the histograms are applied based on super-pixelneighborhoods. Let G(S, E) be the adjacency graph of super-pixels s_(i)in an image, and h_(i) ⁰ be the non-normalized histogram associated withthis region. E is the set of edges formed between pairs of adjacentsuper-pixels (s_(i), s_(j)) in the image and D(s_(i), s_(j)) is thelength of shortest path between two super-pixels. Then, h_(i) ^(N) a isthe histogram obtained by merging the histograms of the super-pixels_(i) and neighbors who are less than N nodes away in the graph:

$\begin{matrix}{h_{i}^{N} = {\sum\limits_{s_{i}|{{D{({s_{i},s_{j}})}} \leq N}}\; h_{j}^{0}}} & (2)\end{matrix}$The training framework is unchanged, except that super-pixels aredescribed by the normalized histograms h_(i) ^(N) in place of h_(i).

Finally, these 1D merged histogram representations are taken as theinput for the binary SVM training module. After the binary classifier istrained, it is applied to classify all super-pixels from all trainingimages. Then, all the super-pixels labeled as wound are gathered by thefirst stage classifier and an approximately equal number of non-woundsuper-pixels as the training data set for the next stage of machinelearning. For each instance in this set, the dominant color descriptor(DCD) is extracted and train the second stage classifier (whichinherently shares the same working scheme with the first stage SVM basedclassifier) based on these descriptors.

In order to compute this descriptor, the colors present in a givenregion are first clustered. This results in a small number of colors andthe percentages of these colors are calculated. As an option, thevariances of the colors assigned to a given dominant color may also becomputed. The percentages of the colors present in the region should addup to 1. A spatial coherency value is also computed that differentiatesbetween large color blobs versus colors that are spread all over theimage. The descriptor is thus defined as following:F={(c _(i) ,p _(i) ,v _(i)),s},(i=1,2, . . . ,N)  (3)where c_(i) is the i^(th) dominant color and p_(i) is its percentagevalue and v_(i) is its color variance. N represents the number ofdominant color clusters. The spatial coherency s is a single number thatrepresents the overall spatial homogeneity of the dominant colors in theimage. In one instance, the DCD can be easily determined from the earlymean shift based super-pixel segmentation results. The reason for thesecond stage classification is to utilize the color features to furtherimproving the differentiation between skin and wound tissues near thewound boundary.

In the testing process, for an input testing image, the same super-pixelsegmentation and BoW representation generation will be performed. Then,the first stage binary classifier is applied to identify all “candidatewound” super-pixels. Next, the DCD descriptor is generated for each“candidate wound” super-pixel and input to the second stage binaryclassifier. Next, a conditional random field (CRF) technique basedrefinement method is operated to recover more precise boundaries whilestill maintaining the benefits of histogram merge over the super-pixelneighborhood. Finally, a closing operation, one of the morphologymethods, can be performed to eliminate small holes in the detected woundarea and further to smooth the wound boundary.

To train the classifier and also evaluate the wound recognitionperformance of the method of these teachings, the help of experiencedwound clinicians is needed to generate the ground truth wound labels. Inone instance, 48 wound images collected from UMass Wound Clinic from 12patients over 12 months are used. For each image, three clinicians wereasked to delineate the wound boundary independently with Photoshopsoftware and a set of electronic drawing pen and panel. Afterwards, themajority vote scheme is used (for each pixel, if 2 or 3 clinicians labelit as “wound”, then it will be determined as “wound” pixel. Otherwise,it will be determined as “non-wound” pixel). An example of the groundtruth generation is illustrated in FIG. 6.

The samples of the wound recognition results on the images of realpatients are shown in FIG. 7. It can be seen that this solution providepromising wound boundary determination.

In order to better assess the SVM based wound recognition method, thefollowing testing and evaluation approach is used. First, theleave-one-out cross validation method is adopted to evaluate the modelperformance on the entire dataset. Specifically, one image is choseneach time from the sample image set as the testing sample and the restis taken as the training samples used for SVM based model training.Hence, this experiment has to performed for a number of times equal tothe size of the entire sample image set (48 times for all 48 woundimages) in order to test on the entire image dataset and keep thespecified testing image different from all images in the trainingdataset.

Second, since the wound recognition is a skewed distributed binary classproblem, which contains a large number of non-wound super-pixels and arelatively small number of wound super-pixels for each wound image, theaccuracy rate cannot be used to evaluate the performance. Instead, theidea of true positive (tp), false positive (fp), false negative (fn) andtrue negative (tn), respectively, defined as in FIG. 8 is used. Thematrix shown in this figure is also called the confusion matrix or anerror matrix.

Substantial research has been performed on developing a convincingevaluation score based on these four values. The Matthews CorrelationCoefficient (MCC) is used in machine learning as a measure of thequality of binary classification. Especially, it takes into account trueand false positives and negatives and is generally regarded as abalanced measure which can be used even if the classes are of verydifferent sizes.

The MCC is in essence a correlation coefficient between the observed andpredicted binary classification; it returns a value between −1 and +1. Acoefficient of +1 represents a perfect prediction, 0 no better thanrandom prediction and −1 indicates total disagreement between predictionand observation. It is defined directly on the confusion matrix asbelow.

$\begin{matrix}{{MCC} = \frac{{{tp} \times {tn}} - {{fp} \times {fn}}}{\sqrt{\left( {{tp} + {fp}} \right)\left( {{tp} + {fn}} \right)\left( {{tn} + {fp}} \right)\left( {{tn} + {fn}} \right)}}} & (4)\end{matrix}$The experimental results shows that the average MCC value of 48 testimages using leave-one-out evaluation method is 0.7, which is 0.1 higherthan the commonly regarded standard value of promising objectrecognition.

CRFs are essentially a way of combining the advantages of discriminativeclassification and graphical modeling, combining the ability tocompactly model multivariate outputs y with the ability to leverage alarge number of input features x for prediction (see, for example, C.Sutton and A. McCallum, An Introduction to Conditional Random Fields,Foundations and Trends in Machine Learning, Vol. 4, No. 4 (2011)267-373, which is incorporated by reference herein in its entirety andfor all purposes). the conditional random field (CRF) based models havebeen widely applied to object classification (image labeling) tasks dueto its generative nature and flexibility to incorporate various featuresin a single unified formulation.

A number of models have been proposed in recent years (see P.Krahenbuhl, V. Koltun, and P. Krahenbuhl, “Efficient Inference in FullyConnected CRFs with Gaussian Edge Potentials,” Adv. Neural Inf. Process.Syst. 24 (Proceedings NIPS), no. 4, pp. 1-9, 2011, X. He, R. S. Zemel,and M. a. Carreira-Perpinan, “Multiscale conditional random fields forimage labeling,” in Proceedings of the 2004 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition, 2004, vol. 2, pp.695-702, J. Malik, S. Belongie, T. Leung, and J. Shi, “Contour andtexture analysis for image segmentation,” Int. J. Comput. Vis., vol. 43,no. 1, pp. 7-27, 2001, all of which are incorporated by reference hereinin their entirety and for all purposes). The major difference betweenmodels lies with the potential granularity (pixel-wise or super-pixelwise) and the features used to generate the potential term.Consequently, the inference algorithm for each model varies accordingly.In this section, we will introduce three CRF based models, each of whichhas been claimed to provide strong performance in object classificationon natural scene images (see System Designs for Diabetic Foot UlcerImage Assessment, Ph. D. Dissertation by Lei Wang, submitted to theDepartment of Electrical and Computer Engineering, WORCESTER POLYTECHNICINSTITUTE, February 2016, which is incorporated by reference herein inits entirety and for all purposes).

In most of recent CRF models for image labeling, the unary potentialsfor pixel-wise features are derived from TextonBoost, which estimatesthe probability distribution of labels on current pixel by boosting weakclassifiers based on a set of shape filter responses J. Shotton, J.Winn, C. Rother, and a Criminisi, “{TextonBoost} for ImageUnderstanding: Multi-Class Object Recognition and Segmentation byJointly Modeling Appearance, Shape and Context,” vol. 81, no. 1, pp.2-23, 2009, which is incorporated by reference herein in its entiretyand for all purposes). Before discussing each CRF model, we willintroduce the TextonBoost concept at first. A general TextonBoostprocess is shown in FIG. 5 a.

This section describes the first two functional blocks in FIG. 5a .Efficiency demands a compact representation for the range of differentappearance of an object. Textons (J. Malik, S. Belongie, T. Leung, andJ. Shi, “Contour and texture analysis for image segmentation,” Int. J.Comput. Vis., vol. 43, no. 1, pp. 7-27, 2001) have been used since theyhave been proven effective in categorizing materials as well as genericobject classes. The term texton was utilized for describing humantextural perception, and is somewhat analogous to phonemes used inspeech recognition.

An example of a standard textonization process is described below. Forthe first step, the training images are convolved with a filter bank atdifferent scales. There are actually quite a number of different optionsfor the filter bank, where the only requirement is that the filter bankshould be sufficiently representative. In this work, we apply the samefilter-bank as in J. Winn, A. Criminisi, and T. Minka, “Objectcategorization by learned universal visual dictionary,” Tenth IEEE Int.Conf. Comput. Vis. Vol. 1, vol. 2, pp. 1800-1807, 2005, incorporated byreference herein in its entirety and for all purposes, which consists ofGaussians at three different scales (σ=1, 2, 4), Laplacian of Gaussians(LoG) at four different scales (σ=1, 2, 4, 8) and 4 first orderderivatives of Gaussians (DoG). The Gaussians are applied to all threecolor channels, and then produce 9 filter responses. The LoGs areapplied only to the luminance channel in CIE Lab space and provide 4filter responses. Finally, DoGs, with two different scales and twodifferent directions, are also only applied to the luminance channel andgive 4 filter responses. Hence, 17 dimensional filter responses areproduced in total for each pixel.

The original RGB color space needs to be converted to CIE Lab space forperceptual uniformity. This filter-bank was determined to have full rankin a singular-value decomposition and therefore contains no redundantelement. The 17 dimensional responses for all training pixels are thennormalized (to give zero mean and unit covariance), and an unsupervisedclustering is performed to generate the texton dictionary. Asrecommended by previous works, we apply the Euclidean-distance K-meansclustering algorithm. The one most obvious shortcoming of the originalK-means algorithm is its computational expense. Fortunately, its timeperformance can be greatly improved by employing the triangle inequalitytechniques for acceleration. Finally the texton maps are generated byassigning each pixel in each image to the nearest cluster center. Wewill denote the texton map as T where pixel i has value T_(i) E (1, 2, .. . K) and where K represents the cluster number set in the K-meanalgorithm. In practice, we can use other dense features, such as thelocation feature, instead of the filter bank output introduced above.Actually, we extract multiple features at the same time for each pixeland generate an independent texton map based on each feature.

Each texture-layout filter is a pair (r, t) of an image region r and atexton t. Region r is defined in coordinates relative to the pixel ibeing classified. For simplicity, a set R of candidate rectangles arechosen at random, such that their top-left and bottom right corner liewithin a fixed bounding box covering about half the image area. Thebounding box was ±100 pixels in x and y direction. This enables themodel to involve long-range contextual information, in addition to theoriginal CRF model which only contains pixel-wise connections betweenadjacent pixels in the second order clique.

The unary potential term based on texture-layout filter output wastrained using the adapted version of the Joint Boost algorithm (A.Torralba, K. P. Murphy, and W. T. Freeman, “Sharing visual features formulticlass and multiview object detection,” IEEE Trans. Pattern Anal.Mach. Intell., vol. 29, no. 5, pp. 854-869, 2007, which is incorporatedby reference herein in its entirety and for all purposes) which combineda number of “weak classifiers” (iteratively selected discriminativetexture-layout filters) into a strong classifier P(c|x, i). Each weakclassifier would be shared by a number of classes (class set C). In thiscase, each weak classifier is capable of dealing with themulti-classification task between the classes in C. This also gives usthe possibility of more efficient classification and bettergeneralization.

The first CRF model we will apply to our wound recognition task wasproposed in [99]. The CRF energy formulation is shown as below.

$\begin{matrix}{{{E(y)} = {{\sum\limits_{i}\;\overset{\overset{{unary}\mspace{14mu}{potential}\mspace{14mu}{terms}}{︷}}{\left( {{\varphi_{i}\left( {y_{i},x_{i},\theta_{\varphi}} \right)} + {\pi\left( {y_{i},x_{i},\theta_{\pi}} \right)} + {\lambda\left( {y_{i},x_{i},\theta_{\lambda}} \right)}} \right)}} + {\sum\limits_{i,{j \in N},}\;\overset{\overset{{pairwise}\mspace{14mu}{potential}}{︷}}{\phi\left( {y_{i},y_{j},{g_{ij}\left( {x_{i},x_{j}} \right)},\theta_{\phi}} \right)}} - {\log\;{Z\left( {x,\theta} \right)}}}}\mspace{20mu}{{\varphi_{i}\left( {y_{i},x_{i},\theta_{\varphi}} \right)} = {p\left( {\left. y_{i} \middle| x_{i} \right.,\theta_{\varphi}} \right)}}} & \left( {4a} \right)\end{matrix}$is the unary potential term derived from the texture-layout boostedclassifier trained by TextonBoost method. This term incorporates thetexture, layout, and textural context information of the object classes.This unary term is the most powerful term in this CRF model.π(y _(i) ,x _(i),θ_(π))is the unary color potential term which derived from the GaussianMixture Models (GMMs) in CIE Lab color space, and the mixturecoefficients depend on the class label. The conditional distribution ofthe color x given a pixel depending on a class label c is shown in theformula below.

${p\left( x \middle| y \right)} = {\sum\limits_{k}\;{{p\left( x \middle| k \right)}{p\left( k \middle| y \right)}}}$p(x|k) = N(x|μ_(k), Σ_(k))where p(x|k) is the component Gaussian density for each cluster. Eachcomponent density is a multi-dimensional Gaussian function, where μk andΣ_(k) are the mean and covariance matrix for cluster center k. Thedistribution term p(k|y) can be viewed as the mixture weight(coefficient) for cluster center k. The color potential is as below.

${\pi\left( {y_{i},x_{i},\theta_{\pi}} \right)} = {\log\;{\sum\limits_{k}\;{{\theta_{\pi}\left( {y_{i},k} \right)}{p\left( k \middle| x_{i} \right)}}}}$

Comparing the two equations above), we can find that the parameter termθ_(π.)(y_(i),k) represent the distribution term p(k|y) in eq. (5.22) fori^(th) label in the label set. The term p(k|x_(i))∝p(x_(i)|k) based onBayesian rule given the known prior probability of the cluster centerp(k). Now the only task left for us is to learn the parameter. First,the color clusters are learned in an unsupervised manner using K-means.The iterative algorithm, called Expectation and Maximization (EM), thenalternates between inferring the optimal labeling (expectation step) andcomputing the parameters for potential term (maximization step). Thedetails about the EM algorithm can be found in A. P. Dempster, N. M.Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via theEM algorithm,” J. R. Stat. Soc. Ser. B, vol. 39, no. 1, pp. 1-38, 1977,which is incorporated by reference herein in its entirety and for allpurposes).

Basic CRF models consist of unary potentials, defined on individualpixels, and pair-wise potential terms defined on pairs of adjacentpixels. By incorporating smoothness term in the CRF potentials, similarpixels are encouraged to have the same label, and the contextualrelationship between different object classes can be modeled. Accordingto L. Ladický, C. Russell, P. Kohli, and P. H. S. Torr, “Associativehierarchical random fields,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 36, no. 6, pp. 1056-1077, 2014, which is incorporated by referenceherein in its entirety and for all purposes, the nature of the adjacencybasic CRF structure results in the inability to include long rangeconnections within image. Consequently, the inaccurate classification islikely to happen at the object boundary due to excessive smoothness.

To solve this problem, higher order potentials, defined on super-pixelsor between pair of super-pixels, were incorporated into the basic CRFmodels to better describe the hierarchical connectivity. This methodgives us a integration of the “top-down” and “bottom-up” approaches thatare common to many problems in computer vision. To achieve thisunification, a smart model, called the associative hierarchical randomfield (AHRF) was proposed in L. Ladický, C. Russell, P. Kohli, and P. H.S. Torr, “Associative hierarchical random fields,” IEEE Trans. PatternAnal. Mach. Intell., vol. 36, no. 6, pp. 1056-1077, 2014. Moreimportantly, it is shown that this model can be solved efficiently usinggraph-cut based move making algorithms mentioned earlier. It was alsoproved that a new model generated by summing up two AHRFs is also anAHRF which can be solved effectively. It enables different potentialsbased on different features to be incorporated within the CRF model andthe model inference is still practical.

Although the hierarchical connectivity and high-order potentials definedon super-pixel are incorporated into the CRF framework, some researchersclaim that these methods suffer from the instability of unsupervisedsuper-pixel segmentation algorithms, especially with respect to theability to recognize objects with complicated boundaries (P. Krahenbuhl,V. Koltun, and P. Krahenbuhl, “Efficient Inference in Fully ConnectedCRFs with Gaussian Edge Potentials,” Adv. Neural Inf. Process. Syst. 24(Proceedings NIPS), no. 4, pp. 1-9, 2011). A fully connected CRF modelhas been proposed to refine the image labeling results. In the fullyconnected model, each pair of pixels in the image is connected by anedge, which has been further associated with a pairwise potential. Themain challenge for this model is the size of this model: even for a lowresolution image, there are in the order of 10⁶ of nodes and 10¹⁰ ofedges. To deal with this gigantic problem, a highly efficient inferencealgorithm has been proposed. In this approach, the pairwise edgepotentials are defined by a linear combination of Gaussian kernels in anarbitrary feature space. The CRF distribution was estimated by a meanfield approximation (L. E. P. Xing, S. P. Schulam, and W. Wang, “13:Variational Inference: Loopy Belief Propagation and Mean Field,” no. 1,pp. 1-10, 2012, which is incorporated by reference herein in itsentirety and for all purposes). Most importantly, it was proved that amean field update of all variables in a fully connected CRF can beperformed using Gaussian filtering in feature space. This inferencealgorithm is sub-linear in the number of edges in the model.

Hereinbelow, a design of a wound classifier based on the CRF modelsdescribed. The entire model training and wound recognition process isillustrated in FIG. 5b . Different CRF models correspond to differentunary and pairwise potential terms.

The wound classifier training process is shown in the left column inFIG. 5a . Most modules in the system have been introduced hereinabove.There are many different algorithms for super-pixel segmentation. In oneembodiment, the parallel version of SLIC algorithm (R. Achanta, A.Shaji, and K. Smith, “SLIC Superpixels Compared to State-of-the-ArtSuperpixel Methods,” Pattern Anal., vol. 34, no. 11, pp. 2274-2281,2012, which is incorporated by reference herein in its entirety and forall purposes) is applied due to its good boundary adherence andefficient implementation. Note that there is no need for super-pixelsegmentation and super-pixel based potential training if CRF model 1 or3 described hereinabove has been applied.

Color based features in various color spaces have proven to be the mosteffective discriminative indicators for wound classification against thehealthy skin. As mentioned hereinabove, other texture based featureshave also been adopted as auxiliary tools. In our classifier, we extractfilter-bank based features, dense SIFT (DSIFT) feature (B. Fulkerson, A.Vedaldi, and S. Soatto, “Class segmentation and object localization withsuperpixel neighborhoods,” in Computer Vision, 2009 IEEE 12thInternational Conference on, 2009, which is incorporated by referenceherein in its entirety and for all purposes) location features andopponent SIFT feature for boosting the unary potential term. The woundrecognition system of these teachings is expected to be able todetermine the accurate wound boundary from less controlled images wherethe illumination and angles can vary. Especially, these images mightcontain wounds acquired at different ranges, and the images may alsocontain other background objects. Therefore, we apply these featureswhich have already provided promising performance on objectclassification tasks in natural scene images. The features used forunary potential term training are all supposed to be extracted densely,i.e. the feature vector is extracted at every pixel location for alltraining images. However, this is not practical considering the largevolume of dataset and high resolution of images. Instead, we extractfeatures on a down-sampled version of the original image grid. Thedown-sampled rate for each feature has been determined empirically.

There are two different ways to incorporate these features into the CRFmodel. In the first method, we can further decompose the unary potentialterm as a weighted summation

${{\varphi(x)} = {\sum\limits_{c}\;{\lambda_{c}{\xi_{c}(x)}}}},$where ξ_(c)(x) is a feature-based potential and λ_(c) is its weight. Weneed to perform the joint boost approach to learn each feature basedpotential, then we estimate the weights using local search scheme on avalidation set. This training method turns out to be robust, but timeconsuming as well.

The second method to learn a single unary potential term is implementedby combining multiple dense extracted features together. Afterextracting each feature over the image grid, we perform the textongeneration. In this case, we have NM texton channels in total where N isthe number of types of features in total and M is the cluster centernumber for texton generation. Before performing the texture-layoutfiltering, we calculate the integral image, for each channel. Then weextract the texture layout based features based on these NM textonchannels. Finally, we perform the joint-boost approach to learn thefinal unary term only for one time. Weighing the strengths andweaknesses of the first and the second method, we have chosen to applythe second method.

For the pairwise potential terms, no matter which formulation we areusing, the parameters of the model are manually selected to minimize theerror on the validation set using grid search approach (C. J. C. Burges,“A Tutorial on Support Vector Machines for Pattern Recognition,” DataMin. Knowl. Discov., vol. 2, pp. 121-167, 1998, C.-J. L. Chih-Wei Hsu,Chih-Chung Chang, “A Practical Guide to Support Vector Classification,”BJU Int., vol. 101, no. 1, pp. 1396-400, 2008, both of which areincorporated by reference herein in its entirety and for all purposes).

For evaluating the CRF methods' ability to recognize a wound anddetermine its boundary on a given set of images, the super-pixelsegmentation and feature extraction are the same as is used in thetraining process. We applied the learned textons to generate the textonmap for each feature channels. Afterwards, we evaluate the unarypotential, pairwise potential and segment based potentials (ifapplicable) based on the model learned in the training process. Then weapply the CRF inference method to find the optimal labeling over theentire wound image.

To evaluate the performance of the wound recognition system disclosedhereinabove, we apply the three CRF models, disclosed hereinabove, totwo different wound image datasets. The first image dataset is composedof images of Moulage wounds placed on an artificial foot. The seconddataset consists of images of real diabetic foot ulcers from recruitedsubjects at the Wound Clinic in UMass Medical School. To better evaluateour system, the wounds in images of the first dataset were captured atdifferent ranges, illumination levels and viewpoints. Specifically, wecollected 162 images of 6 Moulage wound for the first dataset. 27 imagesfor each wound, at 3 different ranges, 3 different viewpoints, and 3different illumination conditions, were captured. In the second trainingdataset, 100 images were captured for 18 subjects and most of them wereacquired using the image capture box designed in earlier chapter.

To evaluate the performance of the wound recognition over the entiredataset, we split both dataset equally into 10 folders. Then aten-folder validation method is carried out as follows. We will performthe “train and test” operation for 10 rounds. In each round, we trainthe model on 9 folders and test the model on the remaining folder. Thegeneral specificity and sensitivity are evaluated by combining thetesting results from 10 rounds. For the Moulage image dataset, we labelthe image using 4 different labels: the wound, gel which is thetransparent material that surrounds the Moulage wound, the healthy skinand the background. For the real image dataset, the image is labeledinto 3 labels, which are the same as the 4 labels except the surroundinggel category.

To compare the three CRF models, we apply these models one by oneindependently on the same two datasets in the ten-folder validationapproach mentioned above. The two most important parameters are thecluster center N for the texton generation and the boosting iterationnumber M for joint boost training scheme. To obtain better parameterestimation, we perform a grid search method to select the best parameterpair (N, M). We perform the CRF model 3 on the moulage image datasetusing the ten-fold validation method mentioned above. The MatthewsCorrelation Coefficient results are shown as in Table 5.1. And the woundrecognition computation time evaluation results are shown in Table 5.2.We didn't evaluate the training efficiency evaluation since the modeltraining is supposed to be performed “offline”.

TABLE 1 Matthew Correlation Coefficient results using different (N, M)parameter settings N = 100 N = 200 N = 300 N = 400 N = 500 N = 600 M =1000 0.393 0.438 0.471 0.523 0.532 0.538 M = 2000 0.469 0.498 0.5470.596 0.602 0.606 M = 3000 0.550 0.582 0.617 0.648 0.651 0.655 M = 40000.598 0.632 0.668 0.699 0.694 0.703 M = 5000 0.707 0.738 0.769 0.8130.816 0.821

TABLE 2 Wound recognition time using different (N, M) parameter settings(Unit: seconds) N = 100 N = 200 N = 300 N = 400 N = 500 N = 600 M = 100010.2 10.3 10.8 11.0 11.2 11.3 M = 2000 18.1 18.5 19.9 20.2 22.0 22.4 M =3000 27.7 28.2 30.0 30.9 31.3 33.0 M = 4000 38.8 39.9 41.3 41.5 42.942.9 M = 5000 46.3 47.2 49.3 50.1 50.5 51.2

Based on the results shown in Table 5.1 and 5.2, when N=600 and M=5000,the MCC result is the best. Moreover, we can see that the MCC valueincreases as we increase the boosting iteration number, but the timeperformance decreases obviously. On the other hand, when the clustercenter number N becomes larger than 400, there is no obvious improvementfor the MCC result. However, increasing the cluster center willsubstantially increase the computation burden for the model training. Inconclusion, we set N=400 and M=3000 empirically for the best tradeoff ofaccuracy and efficiency.

The specificity and sensitivity evaluation results for the three CRFmodels on two dataset are shown in Table 5.3 and 5.4. Finally, the timeperformance results for wound recognition are shown in Table 5.5. Model1 didn't perform the wound recognition very well on multi-scalesituation, since it is a pairwise model where the pairwise potentialterms have only been evaluated on pair of pixels in the same clique.Model 3 out-performed Model 1 on wound recognition accuracy since itgenerated the pairwise potentials on each pair of pixels in the image.In this case, the long-range connections were incorporated into the CRFformulation. The CRF Model 2 provided even better wound recognitionperformance than Model 3, i.e. the best of the three models introducedin this chapter, especially dealing with images of the same woundcaptured in different ranges (scales), viewpoints and illuminationconditions, due to its hierarchical structure involving super-pixelbased higher-order potential terms. The potentials defined over athree-level hierarchy provide the best tradeoff between the timeperformance and recognition performance, although the hierarchy can beextended indefinitely. It is also found that performance saturated whenthe number of hierarchy level increases beyond three. However, the Model2 required longer computing time than other two models due to thesuper-pixel segmentation and more potential terms to be evaluated. Thecomparison of these three models in terms of specificity and sensitivityis shown in Table 5.3.

TABLE 3 Wound recognition specificity using different CRF models on twodatasets CRF model 1 CRF model 2 CRF model 3 Dataset 1 0.927 0.992 0.984Dataset 2 0.898 0.955 0.911

TABLE 4 Wound recognition sensitivity using different CRF models on twodatasets CRF model 1 CRF model 2 CRF model 3 Dataset 1 0.674 0.844 0.767Dataset 2 0.618 0.769 0.703

TABLE 5 Wound recognition computation time using different CRF models ontwo datasets (Unit: seconds) CRF CRF CRP SVM based model 1 model 2 model3 method Dataset 1 36.7 57.4 30.9 7.3 Dataset 2 37.4 60.3 35.9 8.4

The comparison between the results presented in Table 5.3-5.5 andresults hereinabove indicates that the CRF model based methods are abetter option than the super-pixel based SVM classifier for woundboundary determination tasks with relaxed image capture constraints.However, the SVM classifier based approach is far more computationallyefficient than the CRF model based method.

In one instance, in the method of these teachings, performing the colorsegmentation comprises performing a K-mean color clustering algorithm;and evaluating the wound area comprises using a red-yellow-blackevaluation model for evaluation of the color segmentation and, in thesystem of these teachings, the image segmentation component isconfigured to perform a K-mean color clustering algorithm; and uses ared-yellow-black evaluation model for evaluation of the colorsegmentation.

After the accurate wound boundary is acquired, the wound area isanalyzed within the boundary using some wound description model. Manymethods for assessing and classifying open wounds require advancedclinical expertise and experience. Specialized criteria have beendeveloped for diabetic foot ulcers. In order to facilitate the woundmanagement performed by patients themselves at home, there is need for asimple classification system that can be universally applied. The RYBwound classification model which was first proposed in the October 1988by J. Z. Cuzzell and C. Blanco provide us a consistent, simple model toevaluate the wound (D. Kransner, Wound Care How to Use theRed-Yellow-Black System, the American Journal of Nursing, Vol. 95 (5),1995, pp. 44-47 which is incorporated by reference herein in itsentirety and for all purposes).

The RYB system classifies the wound as red, yellow, black or mixedtissues which represent the different phases of the tissue on thecontinuum of the wound healing process, respectively. In detail, redtissues are viewed as the inflammatory (reaction) phase, proliferation(regeneration), or maturation (remodeling) phase. On the other hand,yellow tissues stand for the infected or contain slough that aren'tready to heal. At last, black tissues indicate necrotic tissue state,which is not ready to heal either.

Based on the RYB wound evaluation model, the task for wound analysis isequal to clustering all the pixels within the wound boundary intocertain color categories. Therefore, all classical clustering method canbe applied to solve this task.

In data mining, k-means clustering is a method of cluster analysis (see,or example, K-means and Hierarchical Clustering, tutorial slides byAndrew W. Moore, 2001, which is incorporated by reference herein in itsentirety and for all purposes), which aims to partition n observationsinto k clusters in which each observation belongs to the cluster withthe nearest mean. In one instance, all the pixels within the woundboundary can be viewed as observations. The three colors referred in RYBmodel are regarded as clusters. The algorithm is graphically illustratedin FIG. 9 a.

There are several things needed to be further specified.

1) The color difference between the cluster center and the target pixel(expressed as Eu in the flowchart in part a) in FIG. 9a is calculated bythe standard Euclidean color difference in CIE Lab model.

2) Strictly speaking, K-mean algorithm is a NP-hard problem, which isunable to converge to a solution within limited time when the image sizeis large enough. However, the iteration can be terminated when theaverage mean variance of each cluster is smaller than a pre-specifiedthreshold. This heuristic method is expressed as the decision block inpart a) of FIG. 3.4. In part a) of FIG. 9a , the initial centers arechosen randomly. However, in practice, the initial centers may bespecified according to some empirical values such as the Macbeth ColorChecker. By this operation, the converging speed will be increased thusmaking the color clustering process more efficient.

As shown in FIG. 9a , the number of cluster is preset to 3. However, thenumber could be smaller or larger than 3. Some post-processing has to beperformed to the resulting clusters. In the present teachings, only thesituation that the number of clusters is equal to 3 at most isconsidered. Therefore, some two or three clusters may have to becombined if they are close to each other enough, which can be equallyviewed as the mean value difference of the two or three clusters issmaller than a preset threshold.

In another embodiment, in the method of these teachings, performing thecolor segmentation includes using a K-mean color clustering algorithm onresults of images used on training a classifier and, in the system ofthese teachings, the image segmentation component is configured to use aK-mean color clustering algorithm on results of images used on traininga classifier.

A method similar to Bag-of-Words is used in another embodiment of colorsegmentation of these teachings. The flow chart of this embodiment isshown in FIG. 9b . There are two major tasks in this algorithm. In thetraining process, as stated herein above, three experienced woundclinicians were asked to label the wound area using a set of electronicdrawing panel, pen and also Photoshop software on the laptop. First, thecolor vectors are gathered, in CIE Lab color space, for all labeledwound pixels in the sample foot ulcer images. Then, the K-meanclustering algorithm is performed for all color vectors. Instead ofusing the pre-set standard color center vector, a number of color vectorvalues are randomly selected as the initial centers. It turns out thatthe setting of the initial centers has no impact on the final colorclustering results. The only parameter, which needs to be preset is thecluster number. In one instance, it is set to 10, a relatively largecluster number considering the wound tissues usually only contain 3-5obviously distinct color types, since this will provide us a more fineclassification and also is not too time demanding, which means eachpixel is assigned to a cluster center reasonably resembling its owncolor. After the initial clustering, all the cluster centers areanalyzed and several centers with small Euclidean distance in the colorspace are merged into one. This operation can reduce the final clusternumber and form a more representative wound evaluation model. From theten color cluster centers resulted from the K-mean algorithm, only anumber of colors centers (1, 2, 3, 4 in one instance) are quite distinctfrom each other. Those color centers can be regarded as clusters foryellow, white, black and red, respectively. Some of the other colorcenters (in one instance from 5-9) can all be classified as one color(red, in one instance) but with different saturation and hues or mergedinto one of the distinct color centers. The original RYB model isextended to include another cluster center representing the wound tissuein color of white. However, based on clinicians' opinion, the whitewound tissue, which is more like the calluses, should not contribute tothe final wound healing assessment. Hence, the white tissue isconsidered as part of the wound area but is not considered whenperforming the color based wound analysis.

In the segmentation process, the original set of clusters (in number of10) is used and the assignment is made to each wound pixel in thedetermined wound area. After that, the pixels assigned to cluster number5-9 are merged into cluster 4, and the pixels assigned to cluster number10 are merged into cluster 3, since the Euclidean distance in CIE Labcolor space is small enough. The color segmentation results on 5 samplewound images are shown in FIGS. 10a-10i , the original images aredisplayed in 10 a-10 c, and the color segmentation results using theK-mean algorithm alone shown in FIGS. 10d-10f . The results from theabove described color segmentation algorithm in this report can be seenin FIGS. 10g-10i . After comparison, algorithm combining k-means withthe figures selected by the clinicians results in an improvement.

In one instance, in the method of these teachings, evaluating the woundarea includes determining a healing score as a method for quantifying ahealing status of the wound area and, in the system of these teachings,the wound evaluation component is configured to determine a healingscore as a method for quantifying a healing status of the wound area.

One goal of these teachings is to provide more meaningful wound analysisresults to the users, including both the clinicians and diabeticpatients. For clinicians, the wound area size and different color tissuecomposition may be sufficient. They can make their diagnosis based onthese raw data. However, for ordinary patients assumed to be without anyclinical knowledge about wounds, only providing them some absolutenumbers does not give them with much help in understanding their actualwound status. Hence, there is a need to translate the raw data into ameaningful numerical value, like a score in the range of 0-100, wherelarger simply means better. In this report, a numerical wound evaluationvalue called healing score is used. The basis for calculating thehealing score are four indicators: wound area size, red tissue size,yellow tissue size, and black tissue size. As introduced in relatedreferences, the red means granulation, which is probably a positive signfor healing. On the other hand, yellow might represent tissues withinfection and black stands for necrotic tissues. And these are negativesigns for bad healing status. Besides, the shrinking of the entire woundarea certainly is a strong positive evidence of good healing status.Note that since there is no official described clinical correspondencefor the white tissue, only the red, yellow and black tissues areconsidered for the calculation of the healing score and will merge thewhite cluster to the closet one of the three types.

Considering all of the factors above, a healing score calculationformula is provided herein below. The Healing Score formulation hasthree components:

1) A Healing Score based on wound area, which will have an initial scoreof 50, given that the wound area can become larger or smaller

2) A Healing Score based on the color with the wound boundary. Here, theinitial score is not fixed, but will be bounded by the range 0-100, suchthat all red will produce a Healing Score of 100, all black will producea Healing Score 0, and some combination of red, white, yellow and blackwill generate a Healing Score 0<score<100.

3) A composite Healing Score, which will be a weighted average of theHealing Score based on wound area and the Healing Score based on thecolor with the wound boundary. The weight may be constant or may beinfluenced by the size of the wound area.

As stated, the initial value is defined to be 50. Let a_(n) be the woundarea in week n and S_(n) ^(A) be the wound area score in week n. a₀ isthe initial wound area size acquired when the patient use the system forthe first time. Thus, S_(n) ^(A)=ƒ(a₀,a_(n)) and S₀ ^(A)=ƒ(a₀,a₀)=50,where f is supposed to be function taking a_(n) and a₀ as itsparameters.

$\begin{matrix}{{{S_{n}^{A} = {\left( {1 - \frac{a_{n} - a_{0}}{a_{0}}} \right)50}},{a_{n} \leq {2\; a_{0}}}}{{S_{n}^{A} = 0},{a_{n} > {2\; a_{0}}}}} & (5)\end{matrix}$As a_(n) varies from 0 to 2a₀ S_(n) ^(A) decreases linearly from 100 to0. For values of a_(n)>2a₀, S_(n) ^(A)=0. This should be reasonableassumption that once the wound become twice as large as the initial sizethere is no sign of healing at all. The wound area healing score is arelative numerical value which takes the initial wound area size as thereference.

Let S_(n) ^(T) be the Healing Score based on the color with the woundboundary in week n. Similar to a_(n), the ratio of red area, yellow areaand black area are defined, within the wound boundary, as r_(n), y_(n)and b_(n), respectively, and where subscript ‘n’ refers to week n.Clearly, r_(n)+y_(n)+b_(n)=1 in general, and specifically r₀+y₀+b₀=1.

Based on wound evaluation theory, S_(n) ^(T) must be formulated so thatS_(n) ^(T)=100 for r_(n)=1; y_(n)=b_(n)=0 and S_(n) ^(T)=0 for b_(n)=1;r_(n)=y_(n)=0 The following formulation for S_(n) ^(T) is proposed:

$\begin{matrix}{S_{n}^{T} = {\frac{1 + r_{n} - {0.5\; y_{n}} - b_{n}}{2}100}} & (6)\end{matrix}$It is easily verified that S_(n) ^(T) (r_(n)=1; y_(n)=b_(n)=0)=100 andthat S_(n) ^(T)(b_(n)=1; y_(n)=r=0)=0. Consider also the case wherer_(n)=y_(n)=b_(n)=0.333 giving S_(n) ^(T)=41.7.

Let S_(n) be the overall, or composite, Healing Score:S _(n) =w _(A) S _(n) ^(A) +w _(T) S _(n) ^(T)  (7)where w_(A) and w_(T) are weights, such that w_(A)+w_(T)=1. This allowsus to formulate S_(n) asS _(n) =w _(A) S _(n) ^(A)+(1+w _(A))S _(n) ^(T)  (8)A simple (and acceptable) solution is to set w_(A)=0.5. w_(A) does nothave be a constant; instead, w_(A) should have a greater influence whenthe wound is close to being healed and hence the area is small.Specifically, in one instance, w_(A) increases linear from w_(A)=0.5 tow_(A)=1.0, as S_(n) ^(A) increases linearly from 0 to 100. In otherwords,

$\begin{matrix}{{{w_{A} = {0.5 + {\frac{0.5}{100}S_{n}^{A}}}},{giving}}{S_{n} = {{\left\lbrack {0.5 + {0.005\; S_{n}^{A}}} \right\rbrack S_{n}^{A}} + {\left\lbrack {0.5 - {0.005\; S_{n}^{A}}} \right\rbrack S_{n}^{T}}}}} & (9)\end{matrix}$

An example of applying the proposed healing score to evaluate the woundstatus is based on five images. The wound analysis data for these fiveimages are shown in Table 6. After calculation, the healing score forthese four wound images are 82.78, 87.86, 84.75, and 75.59 (the firstimage is viewed as the base reference and not scored). From Image 1 to2, the wound area is shrinking. From Image 2 to 3, only a small sizedecrease of the wound area is observed. Hence, there is also a tinyincrease of the healing score by 4.4 points. From part Image 3 to 4,more surgical sutures were exposed and more yellow tissues occurred. Onthe other hand, the size of the entire wound area didn't change toomuch. Corresponding to this trend, the healing score is nearly 3 pointslower than the previous time. Finally, from part Image 4 to 5, there areextra yellow tissues generated on the outer part of the wound and thered tissues are shrinking. On the other hand, the wound and black tissuearea are decreased in a tiny degree. Hence, the healing score decreasedby nearly 9 points.

TABLE 6 Wound assessment results (area unit: mm²) Image 1 Image 2 Image3 Image 4 Image 5 Healing score 82.78 87.86 84.75 75.59 Wound area1126.57 403.09 293.17 279.34 457.39 Red area 791.21 353.25 282.84 214.95106.46 Yellow area 246.42 39.22 10.33 43.41 324.31 Black area 88.9410.62 0 20.98 26.62

Another embodiment of the healing score is presented herein below. Tocreate a measure of wound healing status, we translate the raw woundassessment results into a numerical value called healing score (s_(n))using eq. (10)-(12). Such a single-valued healing score will providepatients and caregivers with a simple measure of the wound healing orwound deterioration relative to the status at the initial visit. Thisscore can range from 0-10. The larger the score is, the better thehealing status is. The fundamental principle underlying the healingscore design is the Red-Yellow-Black (RYB) evaluation model. Thecalculation of the healing score is described in the 3 steps below.

Step 1: For each patient, a reference score of 5 is assigned to thewound image collected at the first visit to the wound clinic;

Step 2: At each subsequent visit, the weighed area of the wound iscalculated by applying eq. (1), where WA_(n) represents the weightedarea of the wound at the nth visit. RA_(n), YA_(n) and BA_(n) representthe red, yellow and black tissue areas, respectively.[W_(R),W_(Y),W_(N)] is the vector of weights for red, yellow and blacktissue areas, respectively. From clinical observations, changes inyellow and black tissue areas influence the healing status more than dochanges in the red tissue area, which can be expressed asW_(R)<W_(Y)<W_(B). In our case, we empirically determined an appropriateweight vector to be [1, 1.5, 2.5].WA _(n) =W _(R) RA _(n) +W _(Y) YA _(n) +W _(B) BA _(n)  (10)

Step 3: Compute a relative healing score using Eq. 2 to compare WA_(n)with WA₀, the weighted area for the first wound image of the currentpatient. The parameter G is an empirically determined gain value,ranging from 0-1, to control the dynamic range of the healing score suchthat our assessment results match clinicians' judgments.

$\begin{matrix}{S_{n} = {{1 - {\frac{{WA}_{n} - {WA}_{0}}{{WA}_{0}}*G}} = {\left( {1 + G} \right) - {G\frac{{WA}_{n}}{{WA}_{0}}}}}} & (11)\end{matrix}$

We find that the gain values of 0.5 and 0.4 provide similarly goodresults. Choosing G=0.4, we verified that S_(n) ranges from 0 to 1.4 ifwe assume that WA_(n) is bound by 0<WA_(n)<3.5WA₀.

To normalize S_(n) into the range [0, 10], we multiply the expression ineq. (11) by 10/1.4. This results in the final formulation of the healingscore, as given in eq. (12). It is easily verified that the healingscore increases from 0 (wound condition is seriously degraded) to 10(wound is completely healed) as the weighted wound area decreases fromits upper bound (3.5WA₀) to 0.

$\begin{matrix}{S_{n} = {10 - \frac{2.857\;{WA}_{n}}{{WA}_{0}}}} & (12)\end{matrix}$

The healing score is a simple, but useful mathematical construct, whichis applicable to other types of chronic wounds, such as venous ulcers,possibly requiring a parameter adjustment.

To establish a clinical basis (ground truth) with which to compare ourwound area, three experienced wound clinicians outlined the wound areaof the wound images in our database independently, using a tabletcomputer and an electronic pen. Their delineations for a given woundwere combined into one ground truth using a majority vote scheme at thepixel level. To assess the accuracy of the wound area determined by themean shift algorithm relative to the ground truth, we apply the MatthewsCorrelation Coefficient (MCC) (B. W. Matthews, “Comparison of thepredicted and observed secondary structure of T4 phage lysozyme”, J.Biochimica et Biophysica Acta (BBA)—Protein Structure, vol. 405, no. 2,pp. 442-451, 1975), which is commonly used for the evaluation of binaryclassification methods.

To provide clinical validation for our healing score, we asked the samethree clinicians to independently score the foot ulcer healing statusfor each wound image over the range from 0-10. A computer-basedapplication was designed to present wound images to clinicians. Only thefirst image is shown initially, and each click of the ‘Next Image’button brings up a new image for scoring, while retaining the previousimages, as shown in FIG. 10 j.

Furthermore, to ascertain whether the quantitative wound data, inaddition to the wound image itself, can improve the clinicians'assessment of the wound, we ask each of the clinicians to score eachwound image twice. In the first round, only wound images are presented,so that clinicians' scores are based solely on their visualobservations. In the second round, the total wound area and the areas ofthe red, yellow and black tissues within the wound boundary are alsopresented. These two sets of scores from the clinicians are compared tothe scores, generated by the healing score algorithm, by calculating theKrippendorff's Alpha Coefficient (KAC) (F. Hayes, K. Krippendorff,“Answering the call for a standard reliability measure for coding data”,J. Communication Methods and Measures, vol. 1, no. 1, pp. 77-89, 2007).KAC is a statistical measure of the agreement of ratings given by two ormore clinicians. The value of this coefficient ranges in [−∝, 1], wherevalue 1 indicates perfect agreement and value 0 indicates the absence ofagreement. A value less than 0 implies that the disagreements aresystematic and exceed what can be expected by chance. The detailedclinical validation results are presented in the “Result” Section.

To evaluate our wound assessment method, we have involved 12 patientsover a period of one year where each patient was seen over a periodranging from 1 month to 5 months in the Wound Clinic at UMass MedicalSchool, based on an approved IRB protocol. Among the 12 patients, 9 ofthem were monitored over at least 2 consecutive visits (2 visits for 3patients, 3 visits for 4 patients, 4 visits for 1 patient and 6 visitsfor 1 patient). In total, 32 foot ulcer images were collected (onepatient, visiting only once, had foot ulcers on both feet) and 28 imageswere used for the clinical validation of the healing score algorithm.

In FIGS. 4 and 5, we present the wound area and tissue classificationresults for two patients, resulting in two time sequences of foot ulcerimages. The average MCC value is 0.68, based on comparison to the woundarea delineation by clinicians for the 32 images, which is better thanwhat was obtained with the method of the other embodiment disclosedhereinabove (which yielded an average MCC value as 0.403) and betterthan the wound recognition method proposed by others (with an averageMCC value of 0.45). Since the image capture box is used for acquiringthe images, the distance between the foot and the imaging plane isconstant. This enables the pixel-dimensions from the algorithmmeasurements to be converted into actual area units (square millimetersin our case) by simply multiplying the pixel dimensions by a constant.The corresponding actual wound area, areas of different wound tissuesand the healing scores for the two patients are shown in Tables 7 and 8.

TABLE 7 Wound assessment results for patient 1 (area unit: mm²). Forbetter accuracy, 2 significant digits for the healing scores aredisplayed. Image 1 Image 2 Image 3 Image 4 Image 5 Healing score 5 (Ref)7.0 7.4 6.9 7.3 Wound area 928 403 293 279 332 Red area 751 353 283 215126 Yellow area 158 39 10 43 184 Black area 19 11 0 21 22

TABLE 8 Wound assessment results for patient 2 (area unit: mm²) Image 1Image 2 Image 3 Healing score 5 (Ref) 3.9 5.6 Wound area 249 329 253 Redarea 203 247 232 Yellow area 11 82 21 Black area 38 0 0

We utilize the Krippendorff's Alpha Coefficient (KAC) to compare theconsistency of healing score among the three clinicians (also referredto as ‘raters’), both for the case where the clinicians are presentedwith only the wound image and for the case where wound assessment dataare also available. The calculated coefficients are referred to as theinter-rater consistency coefficients. The results are shown in thediagonally symmetrical Table 4 where the top number in each cell is theconsistency coefficient for wound image only while the bottom number isthe coefficient for wound image plus quantitative wound data. We can seethat Clinician 1 and 2 have similar assessment about the wound healingstatus irrespective of whether the quantitative data is presented(KAC>0.8 in Cell (1, 2)). Clinician 3's assessment differs somewhat fromthat of the other 2 clinicians (KAC<0.5 in Cell (1, 3) and the topnumber in Cell (2, 3)). Another finding is that Clinician 2 and 3 agreemore when the wound quantitative data is also presented (KAC>0.6 for thebottom number in Cell (2, 3)). Due to our limited number of cliniciansand wound samples, our preliminary results indicating that adding wounddata can have some influence should be tested with a larger group ofclinicians and additional samples.

Next, the effect of providing quantitative wound data, in addition tothe wound image itself, on the healing scoring of a given clinician (or‘rater’) is evaluated by determining the KAC between the healing scoreswith and without the quantitative wound data presented. The evaluationresults are reflected in the intra-rater data impact coefficients. Thequantitative wound data consists of healing score, total wound area andarea components of red, green and black tissues. The results are givenin Table 5 for the three clinicians, showing a modest, but detectableeffect (0.8<KAC<0.9 for each cell); had there been no effect, KAC wouldbe 1.0. We conclude that adding quantitative data to visual imageappears to result in better and/or more consistent assessments, but withour limited set of observations, we cannot generalize as to whetherthese results would apply in a larger wound sample.

The agreement between the algorithm-based healing score and theclinician-based healing score is measured similarly, using the KAC. Theresults are given in Table 11, where the measured coefficients arecalled the clinical validity coefficients. As with the inter-raterconsistency coefficients, the results are provided for both the casewhere the clinicians see only the wound image (top values) and the casewhere they see both the wound image and the quantitative wound data(bottom data). The values in Table 11 show that our healing scorealgorithm agrees well with Clinician 2 (KAC>0.8 especially whenquantitative wound data is presented) and has an acceptably goodagreement with Clinician 1 (KAC>0.6). The KAC value for the scoringresults from Clinician 3 and our algorithm is less than 0.5, possiblyindicating differences in evaluation criteria.

The actual healing scores for 19 wounds (the wound image for eachpatient's initial visit is the reference image) given by 3 clinicians(for the case where both the wound images and quantitative wound dataare presented) and by the above method of these teachings were compared.The scores given by the method of these teachings are a reliablequantitative indicator of the wound healing trend. Overall the method ofthese teachings provides a promising quantitative assessment thatapproximates well the average score from three clinicians.

TABLE 9 Krippendorff's Alpha Coefficients for the inter-raterconsistency measurements, both for wound image only (top values) and forwound image plus quantitative wound data (bottom values) Clinician 1Clinician 2 Clinician 3 Clinician 1 0.85 0.42 0.80 0.46 Clinician 2 0.850.46 0.80 0.63 Clinician 3 0.42 0.46 0.46 0.63

TABLE 10 Krippendorff's Alpha Coefficients for the intra-rater dataimpact measurements Clinician Number 1 2 3 Intra-rater data 0.81 0.800.86 impact coefficients

TABLE 11 Krippendorff's alpha coefficients for the clinical validitymeasurements, both for wound image only (top values) and for wound imageplus quantitative wound data (bottom values) Clinician Number 1 2 3Clinical validity 0.73 0.68 0.42 coefficients 0.66 0.81 0.46

In one embodiment, the system of these teachings includes an imagingcomponent having a first front surface mirror and a second front surfacemirror, the second front surface mirror being disposed at a right angleto the first front surface mirror, the imaging component beingconfigured such that the body part is positioned above the first andsecond front surface mirrors and away from an axis bisecting the rightangle; and wherein the image acquisition device is positioned above thefirst and second front surface mirrors, away from the axis bisecting theright angle and on an opposite side of the axis bisecting the rightangle from the body part.

To ensure consistent image capture conditions and also to facilitate aconvenient image capture process for patients with type 2 diabetes, animage capture device was designed in the shape of a box. This device isreferred to as “the Image Capture Box”. The image capture box wasdesigned as a compact, rugged and inexpensive device that: (i) allowspatients to both view the sole of their foot on the screen of a devicehaving an image capture components (for example, a handheld portablecommunication device such as, but not limited to, a smartphone) and tocapture an image since the majority of patients' wounds occur on thesoles of their feet, (ii) allows patients to rest their feetcomfortably, without requiring angling of the foot or the image capturecomponent 135 (in one instance, a smartphone camera), as patients may beoverweight and have reduced mobility, and (iii) accommodates imageviewing and capture of left foot sole as well as right foot sole. Toachieve these objectives, two front surface mirrors 115, 125 are used,placed at an angle of 90° with respect to each other, and with thecommon line of contact tilted 45° with respect to horizontal. Aschematic drawing of basic optical principle for foot imaging is shownin FIG. 11. The optical path is represented by straight lines witharrows indicating the direction.

A SolidWorks™ 3D rendering of the image capture box is shown in FIGS.12a-12c . As seen in this figure, the entire box has a rectangulartrapezoid shape. Rectangular openings for placing the foot andsmartphone are cut into the slanted surface, shown in FIG. 12b , whichis at 45° with respect to horizontal. In this case, the patient can resthis/her foot comfortably and view his/her wound on the LCD display ofthe smartphone camera. When using the box, the patient needs to ensurethat the wound is completely located within the opening by simplyobserving the image displayed on the smartphone.

To avoid the ghost image effect associated with normal back surfacemirrors (reflective surface on the back side of the glass), frontsurface mirrors (reflective surface on the front side) are needed, asillustrated in FIG. 12a . The optical paths for both the front surfacemirror and the normal mirror are shown in FIG. 13.

In one embodiment, the image acquisition device, the image analysiscomponent, the image segmentation component and the wound evaluationcomponent of the system of these teachings are comprised in a handheldportable electronic device. In that embodiment, the handheld portableelectronic/communication device includes the image acquisition device,the image acquisition device being configured for capturing an image ofa body part including a wound area, one or more processors, and computerusable media having computer readable code embodied therein that, whenexecuted by the one or more processors, causes the one or moreprocessors to extract a boundary of the wound area, perform colorsegmentation within the boundary of the wound area, wherein the woundarea is divided into a plurality of segments, each segment beingassociated with a color indicating a healing condition of the segmentand evaluate the wound area.

Descriptions of exemplary implementations of a mobile-based system canbe found in, for example, U.S. Publication No. 2012/0190947 to Chon etal., which is incorporated herein by reference in its entirety for allpurposes.

FIG. 14 is a block diagram representation of one embodiment of thesystem of these teachings. Referring to FIG. 14, in the embodiment showntherein, a mobile communication system 280 includes a processor 250 andone or more memories 260. In the embodiment shown in FIG. 14, a camera265, where the camera as an objective lens 267, can also supply thephysiological indicators signal to the mobile communication device 280.The one or more memories 260 have computer usable code embodied thereinthat causes the processor 250 to that causes the processor to extract aboundary of the wound area, perform color segmentation within theboundary of the wound area, wherein the wound area is divided into aplurality of segments, each segment being associated with a colorindicating a healing condition of the segment and evaluate the woundarea. In one or more instances, the computer readable code causes theprocessor 250 to perform the implement the methods describedhereinabove.

The one or more memories 260 represent one embodiment of computer usablemedia having computer readable code embodied therein that causes aprocessor to implement the methods of these teachings. Embodiments ofthe method of these teachings are described hereinabove and the computerreadable code can cause a processor to implement those embodiments.

In the embodiment shown in FIG. 14, the mobile communication device 280also includes an antenna 265 that enables communications through one ormore of a variety of wireless protocols or over wireless networks.

In another embodiment, in the system of these teachings, the imageacquisition device is comprised in a handheld portable electronicdevice; and the image analysis component, and the wound evaluationcomponent are comprised in a computing component. The handheld portableelectronic device, such as that shown in FIG. 14, includes the imageacquisition device, the image acquisition device being configured forcapturing an image of a body part including a wound area, one or moreprocessors, and computer usable media having computer readable codeembodied therein that, when executed by the one or more processors,causes the one or more processors to transmit the image to the computingcomponent.

The computing component could have a structure such as that shown inFIG. 15. Referring to FIG. 15, in the structure shown there in, one ormore processors 155 are operatively connected to an input component 160,which could receive the images transmitted by the handheld portableelectronic/communication device, and to computer usable media 165 thathas computer readable code embodied therein, which, when executed by theone or more processors 155, causes the one or more processors 155 toperform the method of these teachings. The one or more processors 155,the input component 160 and the computer usable media 165 areoperatively connected by means of a connection component 170.

In one instance, the computer readable code embodied in the computerusable media 165 of the computing component causes the one or moreprocessors 155 to receive the image from the handheld portableelectronic device, extract a boundary of the wound area, perform colorsegmentation within the boundary of the wound area, wherein the woundarea is divided into a plurality of segments, each segment beingassociated with a color indicating a healing condition of the segmentand evaluate the wound area.

An exemplary embodiment of the system including a handheld portableelectronic/communication device and a computing component (also referredto as a collaborative or cooperative system, is shown in FIG. 16.Referring to FIG. 16, in the exemplary embodiment shown therein, thehandheld portable electronic/communication device is a smart phone andthe computing device is a laptop. The smartphone will play the role ofclient in the communication scheme. It will accomplish the followingtask sequentially: 1) take the picture of the wound and save on thespecified directory on the SD card; 2) make request to the server andsend the JPEG file of wound image to the laptop by function “postO” inan http library and 3) receive the analyzed image file sent by laptopand display it on the screen. The laptop will be viewed as the serverparty, which will 1) listen to the request of the client and get theimage file sent by the client by “dopost( )” function, 2) accomplish thewound image process to he received wound image and 3) send the JPEG fileof the processed image back to the smartphone (client) as a response.

The following is a disclosure by way of example of a device configuredto execute functions (hereinafter referred to as computing device) whichmay be used with the presently disclosed subject matter. The descriptionof the various components of a computing device is not intended torepresent any particular architecture or manner of interconnecting thecomponents. Other systems that have fewer or more components may also beused with the disclosed subject matter. A communication device mayconstitute a form of a computing device and may at least include acomputing device. The computing device may include an inter-connect(e.g., bus and system core logic), which can interconnect suchcomponents of a computing device to a data processing device, such as aprocessor(s) or microprocessor(s), or other form of partly or completelyprogrammable or pre-programmed device, e.g., hard wired and orapplication specific integrated circuit (“ASIC”) customized logiccircuitry, such as a controller or microcontroller, a digital signalprocessor, or any other form of device that can fetch instructions,operate on pre-loaded/pre-programmed instructions, and/or followedinstructions found in hardwired or customized circuitry to carry outlogic operations that, together, perform steps of and whole processesand functionalities as described in the present disclosure.

In this description, various functions, functionalities and/oroperations may be described as being performed by or caused by softwareprogram code to simplify description. However, those skilled in the artwill recognize what is meant by such expressions is that the functionsresult from execution of the program code/instructions by a computingdevice as described above, e.g., including a processor, such as amicroprocessor, microcontroller, logic circuit or the like.Alternatively, or in combination, the functions and operations can beimplemented using special purpose circuitry, with or without softwareinstructions, such as using Application-Specific Integrated Circuit(ASIC) or Field-Programmable Gate Array (FPGA), which may beprogrammable, partly programmable or hard wired. The applicationspecific integrated circuit (“ASIC”) logic may be such as gate arrays orstandard cells, or the like, implementing customized logic bymetallization(s) interconnects of the base gate array ASIC architectureor selecting and providing metallization(s) interconnects betweenstandard cell functional blocks included in a manufacturer's library offunctional blocks, etc. Embodiments can thus be implemented usinghardwired circuitry without program software code/instructions, or incombination with circuitry using programmed software code/instructions.

Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular tangible sourcefor the instructions executed by the data processor(s) within thecomputing device. While some embodiments can be implemented in fullyfunctioning computers and computer systems, various embodiments arecapable of being distributed as a computing device including, e.g., avariety of forms and capable of being applied regardless of theparticular type of machine or tangible computer-readable media used toactually effect the performance of the functions and operations and/orthe distribution of the performance of the functions, functionalitiesand/or operations.

The interconnect may connect the data processing device to define logiccircuitry including memory. The interconnect may be internal to the dataprocessing device, such as coupling a microprocessor to on-board cachememory or external (to the microprocessor) memory such as main memory,or a disk drive or external to the computing device, such as a remotememory, a disc farm or other mass storage device, etc. Commerciallyavailable microprocessors, one or more of which could be a computingdevice or part of a computing device, include a PA-RISC seriesmicroprocessor from Hewlett-Packard Company, an 80x86 or Pentium seriesmicroprocessor from Intel Corporation, a PowerPC microprocessor fromIBM, a Spare microprocessor from Sun Microsystems, Inc, or a 68xxxseries microprocessor from Motorola Corporation as examples.

The inter-connect in addition to interconnecting such asmicroprocessor(s) and memory may also interconnect such elements to adisplay controller and display device, and/or to other peripheraldevices such as input/output (I/O) devices, e.g., through aninput/output controller(s). Typical I/O devices can include a mouse, akeyboard(s), a modem(s), a network interface(s), printers, scanners,video cameras and other devices which are well known in the art. Theinter-connect may include one or more buses connected to one anotherthrough various bridges, controllers and/or adapters. In one embodimentthe I/O controller includes a USB (Universal Serial Bus) adapter forcontrolling USB peripherals, and/or an IEEE-1394 bus adapter forcontrolling IEEE-1394 peripherals.

The memory may include any tangible computer-readable media, which mayinclude but are not limited to recordable and non-recordable type mediasuch as volatile and non-volatile memory devices, such as volatile RAM(Random Access Memory), typically implemented as dynamic RAM (DRAM)which requires power continually in order to refresh or maintain thedata in the memory, and non-volatile ROM (Read Only Memory), and othertypes of non-volatile memory, such as a hard drive, flash memory,detachable memory stick, etc. Non-volatile memory typically may includea magnetic hard drive, a magnetic optical drive, or an optical drive(e.g., a DVD RAM, a CD ROM, a DVD or a CD), or other type of memorysystem which maintains data even after power is removed from the system.

A server could be made up of one or more computing devices. Servers canbe utilized, e.g., in a network to host a network database, computenecessary variables and information from information in the database(s),store and recover information from the database(s), track informationand variables, provide interfaces for uploading and downloadinginformation and variables, and/or sort or otherwise manipulateinformation and data from the database(s). In one embodiment a servercan be used in conjunction with other computing devices positionedlocally or remotely to perform certain calculations and other functionsas may be mentioned in the present application.

At least some aspects of the disclosed subject matter can be embodied,at least in part, utilizing programmed software code/instructions. Thatis, the functions, functionalities and/or operations techniques may becarried out in a computing device or other data processing system inresponse to its processor, such as a microprocessor, executing sequencesof instructions contained in a memory, such as ROM, volatile RAM,non-volatile memory, cache or a remote storage device. In general, theroutines executed to implement the embodiments of the disclosed subjectmatter may be implemented as part of an operating system or a specificapplication, component, program, object, module or sequence ofinstructions usually referred to as “computer programs,” or “software.”The computer programs typically comprise instructions stored at varioustimes in various tangible memory and storage devices in a computingdevice, such as in cache memory, main memory, internal or external diskdrives, and other remote storage devices, such as a disc farm, and whenread and executed by a processor(s) in the computing device, cause thecomputing device to perform a method(s), e.g., process and operationsteps to execute an element(s) as part of some aspect(s) of themethod(s) of the disclosed subject matter.

A tangible machine readable medium can be used to store software anddata that, when executed by a computing device, causes the computingdevice to perform a method(s) as may be recited in one or moreaccompanying claims defining the disclosed subject matter. The tangiblemachine readable medium may include storage of the executable softwareprogram code/instructions and data in various tangible locations,including for example ROM, volatile RAM, non-volatile memory and/orcache. Portions of this program software code/instructions and/or datamay be stored in any one of these storage devices. Further, the programsoftware code/instructions can be obtained from remote storage,including, e.g., through centralized servers or peer to peer networksand the like. Different portions of the software programcode/instructions and data can be obtained at different times and indifferent communication sessions or in a same communication session. Thesoftware program code/instructions and data can be obtained in theirentirety prior to the execution of a respective software application bythe computing device. Alternatively, portions of the software programcode/instructions and data can be obtained dynamically, e.g., just intime, when needed for execution. Alternatively, some combination ofthese ways of obtaining the software program code/instructions and datamay occur, e.g., for different applications, components, programs,objects, modules, routines or other sequences of instructions ororganization of sequences of instructions, by way of example. Thus, itis not required that the data and instructions be on a single machinereadable medium in entirety at any particular instance of time.

In general, a tangible machine readable medium includes any tangiblemechanism that provides (i.e., stores) information in a form accessibleby a machine (i.e., a computing device, which may be included, e.g., ina communication device, a network device, a personal digital assistant,a mobile communication device, whether or not able to download and runapplications from the communication network, such as the Internet, e.g.,an I-phone, Blackberry, Droid or the like, a manufacturing tool, or anyother device including a computing device, comprising one or more dataprocessors, etc.

For the purposes of describing and defining the present teachings, it isnoted that the term “substantially” is utilized herein to represent theinherent degree of uncertainty that may be attributed to anyquantitative comparison, value, measurement, or other representation.The term “substantially” is also utilized herein to represent the degreeby which a quantitative representation may vary from a stated referencewithout resulting in a change in the basic function of the subjectmatter at issue.

Although these teachings have been described with respect to variousembodiments, it should be realized these teachings are also capable of awide variety of further and other embodiments within the spirit andscope of the appended claims.

The invention claimed is:
 1. A method for assessing chronic wounds andulcers, comprising: capturing an image of a body part including a woundarea; the image being a color image; segmenting the image; determining aboundary of the wound area; performing color segmentation within theboundary, wherein the wound area is divided into a plurality ofsegments, each segment being associated with a color indicating ahealing condition of the segment; and evaluating the wound area; whereina wound evaluation component is configured to determine a quantitativehealing score as a method for quantifying a healing status of the woundarea; wherein the quantitative healing score is a wound evaluationvalue; wherein the quantitative healing score is based on a weightedwound area; the weighted wound area being a weighted sum of red tissuewound area, yellow tissue wound area, and black tissue wound area;wherein, weights being predetermined; the quantitative healing scorealso being based on a ratio of the weighted wound area to a weightedwound area for an initial wound image for a same subject; thequantitative healing score comprising a difference between a firstpredetermined constant and a product of a second predetermined constantand the ratio of the weighted wound area to the weighted wound area forthe initial wound image for the same subject; wherein the method is usedfor assessing chronic wounds and ulcers including chronic foot woundsand ulcers from type 2 diabetes.
 2. The method of claim 1, whereinsegmenting the image comprises performing mean shift segmentation. 3.The method of claim 2, wherein the segmenting comprises merging an oversegmented image into a smaller number of regions.
 4. The method of claim1, wherein determining the boundary of the wound area comprises using aconditional random field method.
 5. The method of claim 1, whereinperforming the color segmentation comprises: performing a K-mean colorclustering algorithm.
 6. The method of claim 1, wherein the body partalso includes a calibration patch; the calibration patch locatedproximate to the wound area and substantially in a same plane as thewound area; the calibration patch comprising a number of concentric,circular areas; and wherein the method further comprises: locating thecalibration patch and the number of concentric, circular areas;determining, from a location of the calibration patch and the number ofconcentric, circular areas, whether the image was acquired at an anglerelative to normal incidence; and correcting, when the image wasacquired at the angle relative to normal incidence, the wound area. 7.The method of claim 6, wherein correcting the wound area comprisesmultiplying an observed wound area by a square of a ratio of a referencerange, the reference range being a range at which a conversion factorbetween wound area in pixels and wound area in mm² has been determined,to a range at which the observed wound area was observed, and dividingby the cosine of the angle relative to normal incidence.
 8. A system forassessing wound, comprising: an image acquisition device configured forcapturing an image of a body part including a wound area; the imagebeing a color image; and one or more processors configured to: segmentthe image into a number of regions; extract a boundary of the woundarea; perform color segmentation within the boundary of the wound area,wherein the wound area is divided into a plurality of segments, eachsegment being associated with a color indicating a healing condition ofthe segment; and evaluate the wound area; wherein a wound evaluationcomponent is configured to determine a quantitative healing score as amethod for quantifying a healing status of the wound area; wherein thequantitative healing score is a wound evaluation value; wherein thequantitative healing score is based on a weighted wound area; theweighted wound area being a weighted sum of red tissue wound area,yellow tissue wound area, and black tissue wound area; wherein, weightsbeing predetermined; the quantitative healing score also being based ona ratio of the weighted wound area to a weighted wound area for aninitial wound image for a same subject; the quantitative healing scorecomprising a difference between a first predetermined constant and aproduct of a second predetermined constant and the ratio of the weightedwound area to the weighted wound area for the initial wound image forthe same subject; wherein the system is used for assessing chronicwounds and ulcers including chronic foot wounds and ulcers from type 2diabetes.
 9. The system of claim 8, wherein segmenting the imagecomprises performing mean shift segmentation.
 10. The system of claim 9,wherein the segmenting comprises merging an over segmented image into asmaller number of regions.
 11. The system of claim 8, whereindetermining the boundary of the wound area comprises using a conditionalrandom field method.
 12. The system of claim 8, wherein performing colorsegmentation within the boundary of the wound area comprises performinga K-mean color clustering algorithm; and uses a red-yellow-blackevaluation model for evaluation of the color segmentation.
 13. Thesystem of claim 8, wherein the body part also includes a calibrationpatch; the calibration patch located proximate to the wound area andsubstantially in a same plane as the wound area; the calibration patchcomprising a number of concentric, circular areas; and wherein the oneor more processors are further configured to: determine a location ofthe calibration patch and the number of concentric, circular areas;determine, from the location of the calibration patch and the number ofconcentric, circular areas, whether the image was acquired at an anglerelative to normal incidence; and  correct, when the image was acquiredat the angle relative to normal incidence, the wound area.
 14. Thesystem of claim 13 wherein correcting the wound area comprisesmultiplying an observed wound area by a square of a ratio of a referencerange, the reference range being a range at which a conversion factorbetween wound area in pixels and wound area in mm² has been determined,to a range at which the observed wound area was observed, and dividingby the cosine of the angle relative to normal incidence.
 15. The systemof claim 8, wherein the system comprises a computing component and theimage acquisition device is comprised in a handheld portable electronicdevice; and wherein the handheld portable electronic device comprises:one or more other processors; and non-transitory first computer usablemedia having computer readable code that, when executed by the one ormore other processors, causes the one or more other processors to:transmit the image to the computing component; and wherein the computingcomponent comprises: the one or more processors; and non-transitorysecond computer usable media having computer readable code that, whenexecuted by the one or more processors, configures the one or moreprocessors.
 16. The system of claim 8 further comprising: an imagingcomponent comprising: a first front surface mirror; and a second frontsurface mirror; the second front surface mirror being disposed at aright angle to the first front surface mirror; wherein the imagingcomponent is configured such that the body part is positioned above thefirst and second front surface mirrors and away from an axis bisectingthe right angle; and wherein the image acquisition device is positionedabove the first and second front surface mirrors, away from the axisbisecting the right angle and on an opposite side of the axis bisectingthe right angle from the body part.